Apr  9 00:00:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:00:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:00:47 rdma06 systemd[1]: Starting Rotate log files...
Apr  9 00:00:47 rdma06 systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 1041 (rsyslogd) on client request.
Apr  9 00:00:47 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr  9 00:00:47 rdma06 systemd[1]: Finished Rotate log files.
Apr  9 00:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:00:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:00:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:02:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:02:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:02:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:02:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:02:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:02:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:02:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:02:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:02:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:03:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:03:06 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:03:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:03:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:03:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:03:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:03:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:03:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:04:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:04:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:04:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:04:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:04:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:04:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:05:46 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:05:46 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:07:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:07:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:07:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:07:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:08:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:08:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:08:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:56 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:08:56 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:10:47 rdma06 rsyslogd[1041]: [origin software="rsyslogd" swVersion="8.2102.0-113.el9_2" x-pid="1041" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr  9 00:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:11:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:11:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:36 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:11:36 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:13:16 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:13:16 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:14:26 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:14:26 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:17:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:21:06 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:51:12 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr  9 00:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 00:54:01 rdma06 dnf[667160]: Updating Subscription Management repositories.
Apr  9 00:54:01 rdma06 dnf[667160]: Unable to read consumer identity
Apr  9 00:54:01 rdma06 dnf[667160]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 00:54:01 rdma06 dnf[667160]: Metadata cache refreshed recently.
Apr  9 00:54:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 00:54:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 00:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 00:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 00:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 00:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 00:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 00:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 01:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 01:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 01:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 01:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 01:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:27:47 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 02:27:47 rdma06 dnf[671004]: Updating Subscription Management repositories.
Apr  9 02:27:47 rdma06 dnf[671004]: Unable to read consumer identity
Apr  9 02:27:47 rdma06 dnf[671004]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 02:27:47 rdma06 dnf[671004]: Metadata cache refreshed recently.
Apr  9 02:27:47 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 02:27:47 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 02:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 02:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 02:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 02:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 02:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 02:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 03:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 03:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 03:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 03:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 03:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681028432.8671] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  9 04:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 04:26:01 rdma06 dnf[675786]: Updating Subscription Management repositories.
Apr  9 04:26:01 rdma06 dnf[675786]: Unable to read consumer identity
Apr  9 04:26:01 rdma06 dnf[675786]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 04:26:02 rdma06 dnf[675786]: Copr repo for fmf owned by psss                 3.0 kB/s | 3.3 kB     00:01
Apr  9 04:26:02 rdma06 dnf[675786]: beaker-AppStream                                 41 kB/s | 3.2 kB     00:00
Apr  9 04:26:02 rdma06 dnf[675786]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr  9 04:26:02 rdma06 dnf[675786]: beaker-BaseOS                                    69 kB/s | 2.7 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-BaseOS-debuginfo                          42 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-CRB                                       47 kB/s | 2.7 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-CRB-debuginfo                             56 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-HighAvailability                          99 kB/s | 2.8 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-HighAvailability-debuginfo                34 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-NFV                                       85 kB/s | 2.7 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-NFV-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-RT                                        61 kB/s | 2.7 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-RT-debuginfo                              52 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-ResilientStorage                          59 kB/s | 2.8 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-ResilientStorage-debuginfo                64 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-SAP                                       62 kB/s | 2.7 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-SAPHANA                                   62 kB/s | 2.7 kB     00:00
Apr  9 04:26:03 rdma06 dnf[675786]: beaker-SAPHANA-debuginfo                         43 kB/s | 1.5 kB     00:00
Apr  9 04:26:04 rdma06 dnf[675786]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  9 04:26:05 rdma06 dnf[675786]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  9 04:26:05 rdma06 dnf[675786]: Metadata cache created.
Apr  9 04:26:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 04:26:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 04:26:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.388s CPU time.
Apr  9 04:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 04:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 04:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 04:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 04:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 04:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 05:53:01 rdma06 dnf[679327]: Updating Subscription Management repositories.
Apr  9 05:53:01 rdma06 dnf[679327]: Unable to read consumer identity
Apr  9 05:53:01 rdma06 dnf[679327]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 05:53:01 rdma06 dnf[679327]: Metadata cache refreshed recently.
Apr  9 05:53:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 05:53:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 05:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 05:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 05:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 05:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 05:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 05:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 06:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 06:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 06:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 06:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 06:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:45:47 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 07:45:47 rdma06 dnf[683896]: Updating Subscription Management repositories.
Apr  9 07:45:47 rdma06 dnf[683896]: Unable to read consumer identity
Apr  9 07:45:47 rdma06 dnf[683896]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 07:45:49 rdma06 dnf[683896]: Copr repo for fmf owned by psss                 1.9 kB/s | 3.3 kB     00:01
Apr  9 07:45:49 rdma06 dnf[683896]: beaker-AppStream                                 47 kB/s | 3.2 kB     00:00
Apr  9 07:45:49 rdma06 dnf[683896]: beaker-AppStream-debuginfo                       43 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-BaseOS                                    69 kB/s | 2.7 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-BaseOS-debuginfo                          35 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-CRB                                       47 kB/s | 2.7 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-CRB-debuginfo                             31 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-HighAvailability                         104 kB/s | 2.8 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-HighAvailability-debuginfo                32 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-NFV                                       85 kB/s | 2.7 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-NFV-debuginfo                             40 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-RT                                        56 kB/s | 2.7 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-RT-debuginfo                              62 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-ResilientStorage                          70 kB/s | 2.8 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-ResilientStorage-debuginfo                64 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-SAP                                      161 kB/s | 2.7 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-SAPHANA                                   90 kB/s | 2.7 kB     00:00
Apr  9 07:45:50 rdma06 dnf[683896]: beaker-SAPHANA-debuginfo                         23 kB/s | 1.5 kB     00:00
Apr  9 07:45:51 rdma06 dnf[683896]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  9 07:45:52 rdma06 dnf[683896]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  9 07:45:52 rdma06 dnf[683896]: Metadata cache created.
Apr  9 07:45:52 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 07:45:52 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 07:45:52 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.397s CPU time.
Apr  9 07:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 07:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 07:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 07:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 07:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 07:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:57:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 08:57:01 rdma06 dnf[686797]: Updating Subscription Management repositories.
Apr  9 08:57:01 rdma06 dnf[686797]: Unable to read consumer identity
Apr  9 08:57:01 rdma06 dnf[686797]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 08:57:01 rdma06 dnf[686797]: Metadata cache refreshed recently.
Apr  9 08:57:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 08:57:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 08:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 08:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 08:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 08:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 08:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 08:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 09:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 09:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 09:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 09:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 09:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681050032.9111] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  9 10:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:36:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 10:36:01 rdma06 dnf[690809]: Updating Subscription Management repositories.
Apr  9 10:36:01 rdma06 dnf[690809]: Unable to read consumer identity
Apr  9 10:36:01 rdma06 dnf[690809]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 10:36:01 rdma06 dnf[690809]: Metadata cache refreshed recently.
Apr  9 10:36:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 10:36:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 10:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 10:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 10:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 10:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 10:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 10:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 11:56:01 rdma06 dnf[694048]: Updating Subscription Management repositories.
Apr  9 11:56:01 rdma06 dnf[694048]: Unable to read consumer identity
Apr  9 11:56:01 rdma06 dnf[694048]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 11:56:02 rdma06 dnf[694048]: Copr repo for fmf owned by psss                 3.1 kB/s | 3.3 kB     00:01
Apr  9 11:56:02 rdma06 dnf[694048]: beaker-AppStream                                 55 kB/s | 3.2 kB     00:00
Apr  9 11:56:02 rdma06 dnf[694048]: beaker-AppStream-debuginfo                       50 kB/s | 1.5 kB     00:00
Apr  9 11:56:02 rdma06 dnf[694048]: beaker-BaseOS                                   155 kB/s | 2.7 kB     00:00
Apr  9 11:56:02 rdma06 dnf[694048]: beaker-BaseOS-debuginfo                          41 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-CRB                                       68 kB/s | 2.7 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-CRB-debuginfo                             68 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-HighAvailability                          68 kB/s | 2.8 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-HighAvailability-debuginfo                45 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-NFV                                       76 kB/s | 2.7 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-NFV-debuginfo                             55 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-RT                                        65 kB/s | 2.7 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-RT-debuginfo                              49 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-ResilientStorage                          33 kB/s | 2.8 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-ResilientStorage-debuginfo                70 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-SAP                                      135 kB/s | 2.7 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-SAP-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-SAPHANA                                   81 kB/s | 2.7 kB     00:00
Apr  9 11:56:03 rdma06 dnf[694048]: beaker-SAPHANA-debuginfo                         43 kB/s | 1.5 kB     00:00
Apr  9 11:56:04 rdma06 dnf[694048]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  9 11:56:04 rdma06 dnf[694048]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  9 11:56:05 rdma06 dnf[694048]: Metadata cache created.
Apr  9 11:56:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 11:56:05 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 11:56:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.386s CPU time.
Apr  9 11:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 11:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 11:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 11:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 11:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 11:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 12:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 12:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 12:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 12:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 12:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:15:46 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 13:15:46 rdma06 dnf[697308]: Updating Subscription Management repositories.
Apr  9 13:15:46 rdma06 dnf[697308]: Unable to read consumer identity
Apr  9 13:15:46 rdma06 dnf[697308]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 13:15:46 rdma06 dnf[697308]: Metadata cache refreshed recently.
Apr  9 13:15:46 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 13:15:46 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 13:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 13:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 13:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 13:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 13:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 13:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 14:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 14:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 14:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 14:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 14:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 15:14:01 rdma06 dnf[702105]: Updating Subscription Management repositories.
Apr  9 15:14:01 rdma06 dnf[702105]: Unable to read consumer identity
Apr  9 15:14:01 rdma06 dnf[702105]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 15:14:02 rdma06 dnf[702105]: Copr repo for fmf owned by psss                 4.1 kB/s | 3.3 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-AppStream                                 69 kB/s | 3.2 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-AppStream-debuginfo                       46 kB/s | 1.5 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-BaseOS                                    76 kB/s | 2.7 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-BaseOS-debuginfo                          38 kB/s | 1.5 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-CRB                                       63 kB/s | 2.7 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-CRB-debuginfo                             85 kB/s | 1.5 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-HighAvailability                          90 kB/s | 2.8 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-HighAvailability-debuginfo                46 kB/s | 1.5 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-NFV                                       79 kB/s | 2.7 kB     00:00
Apr  9 15:14:02 rdma06 dnf[702105]: beaker-NFV-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-RT                                        59 kB/s | 2.7 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-RT-debuginfo                              48 kB/s | 1.5 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-ResilientStorage                          82 kB/s | 2.8 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-ResilientStorage-debuginfo                53 kB/s | 1.5 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-SAP                                       90 kB/s | 2.7 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-SAP-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-SAPHANA                                   81 kB/s | 2.7 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-SAPHANA-debuginfo                         43 kB/s | 1.5 kB     00:00
Apr  9 15:14:03 rdma06 dnf[702105]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  9 15:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:14:04 rdma06 dnf[702105]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  9 15:14:04 rdma06 dnf[702105]: Metadata cache created.
Apr  9 15:14:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 15:14:04 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 15:14:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.396s CPU time.
Apr  9 15:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:39:52 rdma06 kernel: perf: interrupt took too long (6178 > 6177), lowering kernel.perf_event_max_sample_rate to 32000
Apr  9 15:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:38 rdma06 systemd[1]: session-9.scope: Deactivated successfully.
Apr  9 15:49:38 rdma06 systemd[1]: session-9.scope: Consumed 37.446s CPU time.
Apr  9 15:49:38 rdma06 systemd-logind[1064]: Session 9 logged out. Waiting for processes to exit.
Apr  9 15:49:38 rdma06 systemd-logind[1064]: Removed session 9.
Apr  9 15:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 15:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 15:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 15:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 15:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 15:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681071632.9454] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  9 16:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:42:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 16:42:01 rdma06 dnf[705680]: Updating Subscription Management repositories.
Apr  9 16:42:01 rdma06 dnf[705680]: Unable to read consumer identity
Apr  9 16:42:01 rdma06 dnf[705680]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 16:42:01 rdma06 dnf[705680]: Metadata cache refreshed recently.
Apr  9 16:42:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 16:42:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 16:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 16:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 16:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 16:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 16:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 16:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 17:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 17:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 17:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 17:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 17:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:39:47 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 18:39:48 rdma06 dnf[710444]: Updating Subscription Management repositories.
Apr  9 18:39:48 rdma06 dnf[710444]: Unable to read consumer identity
Apr  9 18:39:48 rdma06 dnf[710444]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 18:39:49 rdma06 dnf[710444]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-AppStream                                 60 kB/s | 3.2 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-AppStream-debuginfo                       53 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-BaseOS                                    94 kB/s | 2.7 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-BaseOS-debuginfo                          26 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-CRB                                       60 kB/s | 2.7 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-CRB-debuginfo                             82 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-HighAvailability                          58 kB/s | 2.8 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-HighAvailability-debuginfo                45 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-NFV                                       62 kB/s | 2.7 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-NFV-debuginfo                             53 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-RT                                        55 kB/s | 2.7 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-RT-debuginfo                              70 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-ResilientStorage                          50 kB/s | 2.8 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-ResilientStorage-debuginfo                51 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-SAP                                      125 kB/s | 2.7 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-SAPHANA                                   90 kB/s | 2.7 kB     00:00
Apr  9 18:39:50 rdma06 dnf[710444]: beaker-SAPHANA-debuginfo                         65 kB/s | 1.5 kB     00:00
Apr  9 18:39:51 rdma06 dnf[710444]: beaker-harness                                  1.9 kB/s | 1.3 kB     00:00
Apr  9 18:39:52 rdma06 dnf[710444]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  9 18:39:52 rdma06 dnf[710444]: Metadata cache created.
Apr  9 18:39:52 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 18:39:52 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 18:39:52 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.399s CPU time.
Apr  9 18:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 18:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 18:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 18:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 18:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 18:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 19:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 19:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 19:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 19:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 19:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:08:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 20:08:01 rdma06 dnf[714037]: Updating Subscription Management repositories.
Apr  9 20:08:01 rdma06 dnf[714037]: Unable to read consumer identity
Apr  9 20:08:01 rdma06 dnf[714037]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 20:08:01 rdma06 dnf[714037]: Metadata cache refreshed recently.
Apr  9 20:08:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 20:08:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 20:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 20:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 20:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 20:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 20:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 20:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:00:15 rdma06 sshd[716142]: main: sshd: ssh-rsa algorithm is disabled
Apr  9 21:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:00:18 rdma06 systemd-logind[1064]: New session 10 of user root.
Apr  9 21:00:18 rdma06 systemd[1]: Started Session 10 of User root.
Apr  9 21:00:19 rdma06 systemd[1]: Starting Hostname Service...
Apr  9 21:00:19 rdma06 systemd[1]: Started Hostname Service.
Apr  9 21:00:21 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr  9 21:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:00:49 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr  9 21:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:17:00 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 21:17:01 rdma06 dnf[716883]: Updating Subscription Management repositories.
Apr  9 21:17:01 rdma06 dnf[716883]: Unable to read consumer identity
Apr  9 21:17:01 rdma06 dnf[716883]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 21:17:01 rdma06 dnf[716883]: Metadata cache refreshed recently.
Apr  9 21:17:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 21:17:01 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 21:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:33:50 rdma06 systemd[1]: session-10.scope: Deactivated successfully.
Apr  9 21:33:50 rdma06 systemd-logind[1064]: Session 10 logged out. Waiting for processes to exit.
Apr  9 21:33:50 rdma06 systemd-logind[1064]: Removed session 10.
Apr  9 21:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 21:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 21:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 21:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 21:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 21:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681093232.9291] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr  9 22:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:38:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr  9 22:38:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr  9 22:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:42:47 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr  9 22:42:47 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr  9 22:42:47 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr  9 22:42:47 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr  9 22:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:53:17 rdma06 systemd[1]: Starting dnf makecache...
Apr  9 22:53:18 rdma06 dnf[720792]: Updating Subscription Management repositories.
Apr  9 22:53:18 rdma06 dnf[720792]: Unable to read consumer identity
Apr  9 22:53:18 rdma06 dnf[720792]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr  9 22:53:19 rdma06 dnf[720792]: Copr repo for fmf owned by psss                 2.7 kB/s | 3.3 kB     00:01
Apr  9 22:53:19 rdma06 dnf[720792]: beaker-AppStream                                 61 kB/s | 3.2 kB     00:00
Apr  9 22:53:19 rdma06 dnf[720792]: beaker-AppStream-debuginfo                       34 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-BaseOS                                    74 kB/s | 2.7 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-BaseOS-debuginfo                          42 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-CRB                                       62 kB/s | 2.7 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-CRB-debuginfo                             62 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-HighAvailability                          52 kB/s | 2.8 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-HighAvailability-debuginfo                46 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-NFV                                       88 kB/s | 2.7 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-NFV-debuginfo                             51 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-RT                                        54 kB/s | 2.7 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-RT-debuginfo                              49 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-ResilientStorage                          62 kB/s | 2.8 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-ResilientStorage-debuginfo                55 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-SAP                                      202 kB/s | 2.7 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-SAPHANA                                   58 kB/s | 2.7 kB     00:00
Apr  9 22:53:20 rdma06 dnf[720792]: beaker-SAPHANA-debuginfo                         47 kB/s | 1.5 kB     00:00
Apr  9 22:53:21 rdma06 dnf[720792]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr  9 22:53:22 rdma06 dnf[720792]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr  9 22:53:22 rdma06 dnf[720792]: Metadata cache created.
Apr  9 22:53:22 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr  9 22:53:22 rdma06 systemd[1]: Finished dnf makecache.
Apr  9 22:53:22 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.432s CPU time.
Apr  9 22:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 22:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 22:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 22:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 22:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 22:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr  9 23:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr  9 23:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr  9 23:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr  9 23:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr  9 23:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:00:47 rdma06 systemd[1]: Starting Rotate log files...
Apr 10 00:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:00:47 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 10 00:00:47 rdma06 systemd[1]: Finished Rotate log files.
Apr 10 00:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:30:47 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 00:30:48 rdma06 dnf[724772]: Updating Subscription Management repositories.
Apr 10 00:30:48 rdma06 dnf[724772]: Unable to read consumer identity
Apr 10 00:30:48 rdma06 dnf[724772]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 00:30:48 rdma06 dnf[724772]: Metadata cache refreshed recently.
Apr 10 00:30:48 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 00:30:48 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 00:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:46:45 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr 10 00:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 00:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 00:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 00:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 00:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 00:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:27:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:27:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:30:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:30:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:31:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:31:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:33:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:38:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:38:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:38:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:38:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:39:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:39:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:40:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:40:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:40:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:40:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:41:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:41:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:42:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:42:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:43:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:43:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:44:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:44:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:44:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:44:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:45:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:45:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:46:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:46:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:47:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:47:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:48:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:48:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:49:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:51:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:51:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:52:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:53:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:53:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:53:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:53:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:53:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:53:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:53:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:53:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:53:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:53:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:55:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:55:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:55:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:55:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:56:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:56:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:57:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:57:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:57:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:57:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:58:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:58:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:58:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:58:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 01:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 01:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 01:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 01:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 01:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:00:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:00:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:01:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:01:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:01:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:01:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:01:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:01:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:01:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:01:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:02:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:02:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:02:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:02:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:03:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:03:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:03:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:03:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:04:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:04:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:04:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:04:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:05:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:05:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:06:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 02:06:01 rdma06 dnf[728638]: Updating Subscription Management repositories.
Apr 10 02:06:01 rdma06 dnf[728638]: Unable to read consumer identity
Apr 10 02:06:01 rdma06 dnf[728638]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 02:06:03 rdma06 dnf[728638]: Copr repo for fmf owned by psss                 2.0 kB/s | 3.3 kB     00:01
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-AppStream                                 71 kB/s | 3.2 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-AppStream-debuginfo                       35 kB/s | 1.5 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-BaseOS                                   134 kB/s | 2.7 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-BaseOS-debuginfo                          50 kB/s | 1.5 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-CRB                                       55 kB/s | 2.7 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-CRB-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-HighAvailability                          74 kB/s | 2.8 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-HighAvailability-debuginfo                50 kB/s | 1.5 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-NFV                                      104 kB/s | 2.7 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-NFV-debuginfo                             56 kB/s | 1.5 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-RT                                        57 kB/s | 2.7 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-RT-debuginfo                              62 kB/s | 1.5 kB     00:00
Apr 10 02:06:03 rdma06 dnf[728638]: beaker-ResilientStorage                          60 kB/s | 2.8 kB     00:00
Apr 10 02:06:04 rdma06 dnf[728638]: beaker-ResilientStorage-debuginfo                53 kB/s | 1.5 kB     00:00
Apr 10 02:06:04 rdma06 dnf[728638]: beaker-SAP                                       89 kB/s | 2.7 kB     00:00
Apr 10 02:06:04 rdma06 dnf[728638]: beaker-SAP-debuginfo                             74 kB/s | 1.5 kB     00:00
Apr 10 02:06:04 rdma06 dnf[728638]: beaker-SAPHANA                                   53 kB/s | 2.7 kB     00:00
Apr 10 02:06:04 rdma06 dnf[728638]: beaker-SAPHANA-debuginfo                         36 kB/s | 1.5 kB     00:00
Apr 10 02:06:04 rdma06 dnf[728638]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 10 02:06:05 rdma06 dnf[728638]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 10 02:06:05 rdma06 dnf[728638]: Metadata cache created.
Apr 10 02:06:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 02:06:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 02:06:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.442s CPU time.
Apr 10 02:06:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:06:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:06:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:06:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:07:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:07:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:07:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:07:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:07:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:07:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:08:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:08:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:08:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:09:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:09:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:09:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:09:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:10:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:10:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:10:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:10:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:10:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:10:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:10:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:10:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:10:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:10:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:11:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:11:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:11:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:11:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:11:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:11:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:12:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:12:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:12:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:12:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:12:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:12:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:12:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:12:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:12:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:12:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:12:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:12:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:13:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:13:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:13:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:13:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:13:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:13:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:13:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:13:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:13:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:13:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:14:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:14:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:15:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:15:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:15:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:15:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:16:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:16:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:16:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:16:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:16:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:16:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:17:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:17:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:17:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:17:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:17:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:17:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:18:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:18:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:18:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:18:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:18:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:18:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:18:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:18:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:18:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:18:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:18:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:18:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:19:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:19:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:19:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:19:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:19:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:19:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:19:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:20:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:20:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:20:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:20:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:20:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:20:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:20:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:20:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:20:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:20:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:20:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:20:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:21:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:21:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:21:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:21:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:21:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:21:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:21:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:21:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:22:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:22:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:22:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:22:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:22:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:22:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:22:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:22:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:22:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:22:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:22:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:22:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:23:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:23:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:23:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:23:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:23:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:23:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:23:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:23:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:23:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:23:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:23:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:23:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:24:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:24:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:24:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:24:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:24:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:24:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:24:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:24:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:24:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:25:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:25:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:25:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:25:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:25:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:25:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:25:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:25:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:25:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:25:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:25:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:25:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:26:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:26:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:26:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:26:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:26:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:26:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:26:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:26:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:27:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:27:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:27:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:27:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:27:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:27:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:27:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:27:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:27:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:27:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:28:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:28:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:28:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:28:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:28:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:28:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:28:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:28:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:28:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:28:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:28:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:28:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:29:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:29:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:29:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:29:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:29:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:29:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:29:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:29:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:29:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:29:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:29:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:29:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:30:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:30:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:30:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:30:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:30:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:30:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:30:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:30:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:30:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:30:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:31:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:31:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:31:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:31:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:31:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:31:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:31:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:31:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:31:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:31:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:32:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:32:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:32:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:32:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:32:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:32:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:32:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:32:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:32:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:32:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:32:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:32:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:33:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:33:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:33:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:33:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:33:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:33:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:33:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:33:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:33:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:33:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:34:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:34:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:34:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:34:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:34:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:34:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:34:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:34:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:34:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:34:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:34:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:34:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:35:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:35:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:35:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:35:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:35:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:35:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:35:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:35:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:35:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:35:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:35:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:35:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:36:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:36:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:36:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:36:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:36:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:36:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:36:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:36:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:36:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:36:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:36:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:37:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:37:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:37:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:37:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:37:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:37:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:37:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:37:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:37:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:37:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:37:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:37:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:38:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:38:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:38:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:38:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:38:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:38:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:38:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:38:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:39:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:39:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:39:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:39:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:39:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:39:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:39:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:39:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:39:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:39:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:40:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:40:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:40:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:40:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:40:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:40:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:41:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:41:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:41:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:41:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:41:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:41:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:41:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:41:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:41:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:41:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:42:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:42:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:42:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:42:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:42:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:42:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:42:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:42:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:42:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:42:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:43:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:43:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:43:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:43:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:43:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:43:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:43:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:44:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:44:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:44:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:44:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:44:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:44:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:44:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:44:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:45:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:45:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:45:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:45:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:45:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:45:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:45:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:45:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:45:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:45:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:46:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:46:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:46:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:46:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:46:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:46:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:46:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:46:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:46:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:46:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:47:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:47:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:47:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:47:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:47:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:47:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:47:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:47:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:47:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:47:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:48:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:48:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:48:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:48:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:48:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:48:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:48:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:48:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:48:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:48:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:49:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:49:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:49:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:49:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:49:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:49:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:49:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:49:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:49:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:49:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:50:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:50:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:50:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:50:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:50:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:50:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:50:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:50:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:50:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:50:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:50:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:50:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:51:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:51:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:51:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:51:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:51:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:51:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:51:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:51:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:51:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:51:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:52:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:52:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:52:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:52:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:52:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:52:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:52:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:52:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:52:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:52:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:53:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:53:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:54:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:54:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:54:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:54:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:54:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:54:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:54:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:54:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:54:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:54:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:54:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:54:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:55:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:55:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:55:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:55:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:55:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:55:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:55:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:55:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:56:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:56:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:56:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:56:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:56:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:56:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:56:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:56:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:57:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:57:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:57:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:57:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:57:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:57:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:57:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:57:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:58:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:58:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:58:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:58:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:58:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:58:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:58:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:58:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:59:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:59:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:59:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:59:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:59:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:59:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 02:59:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:59:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 02:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 02:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 02:59:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 02:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:00:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:00:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:00:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:00:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:00:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:00:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:00:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:00:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:00:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:00:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:01:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:01:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:01:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:01:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:02:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:02:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:02:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:02:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:02:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:02:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:02:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:02:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:03:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:03:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:03:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:03:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:03:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:03:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:03:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:03:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:04:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:04:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:04:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:04:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:04:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:04:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:04:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:04:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:05:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:05:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:05:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:05:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:05:24 rdma06 sshd[731068]: main: sshd: ssh-rsa algorithm is disabled
Apr 10 03:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:05:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:05:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:05:33 rdma06 systemd-logind[1064]: New session 11 of user root.
Apr 10 03:05:33 rdma06 systemd[1]: Started Session 11 of User root.
Apr 10 03:05:34 rdma06 systemd[1]: Starting Hostname Service...
Apr 10 03:05:34 rdma06 systemd[1]: Started Hostname Service.
Apr 10 03:05:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:05:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:05:39 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr 10 03:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:05:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:05:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:06:04 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 10 03:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:06:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:06:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:06:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:06:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:06:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:06:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:06:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:06:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:07:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:07:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:07:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:07:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:07:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:07:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:08:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:08:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:08:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:08:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:08:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:08:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:08:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:08:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:09:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:09:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:09:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:09:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:09:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:09:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:09:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:09:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:10:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:10:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:11:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:11:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:11:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:11:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:11:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:11:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 03:12:48 rdma06 dnf[731475]: Updating Subscription Management repositories.
Apr 10 03:12:48 rdma06 dnf[731475]: Unable to read consumer identity
Apr 10 03:12:48 rdma06 dnf[731475]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 03:12:48 rdma06 dnf[731475]: Metadata cache refreshed recently.
Apr 10 03:12:48 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 03:12:48 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 03:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:13:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:13:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:14:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:14:07 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:14:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:14:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:14:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:14:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:14:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:14:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:14:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:14:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:15:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:15:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:15:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:15:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:15:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:15:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:16:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:16:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:16:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:16:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:16:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:16:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:17:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:17:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:17:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:17:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:17:47 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:17:47 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:19:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:19:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:19:27 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:19:27 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:19:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:21:17 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:21:17 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:26:37 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:26:37 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:26:57 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:26:57 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:36:07 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:38:13 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr 10 03:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 03:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 03:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 03:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 03:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 03:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681114832.9416] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 10 04:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 04:41:01 rdma06 dnf[735056]: Updating Subscription Management repositories.
Apr 10 04:41:01 rdma06 dnf[735056]: Unable to read consumer identity
Apr 10 04:41:01 rdma06 dnf[735056]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 04:41:01 rdma06 dnf[735056]: Metadata cache refreshed recently.
Apr 10 04:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 04:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 04:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 04:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 04:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 04:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 04:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 04:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 05:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 05:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 05:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 05:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 05:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:10:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 06:10:01 rdma06 dnf[738661]: Updating Subscription Management repositories.
Apr 10 06:10:01 rdma06 dnf[738661]: Unable to read consumer identity
Apr 10 06:10:01 rdma06 dnf[738661]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 06:10:02 rdma06 dnf[738661]: Copr repo for fmf owned by psss                 3.7 kB/s | 3.3 kB     00:00
Apr 10 06:10:03 rdma06 dnf[738661]: beaker-AppStream                                2.6 kB/s | 3.2 kB     00:01
Apr 10 06:10:03 rdma06 dnf[738661]: beaker-AppStream-debuginfo                       38 kB/s | 1.5 kB     00:00
Apr 10 06:10:03 rdma06 dnf[738661]: beaker-BaseOS                                   125 kB/s | 2.7 kB     00:00
Apr 10 06:10:03 rdma06 dnf[738661]: beaker-BaseOS-debuginfo                          46 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-CRB                                       55 kB/s | 2.7 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-CRB-debuginfo                             49 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-HighAvailability                          33 kB/s | 2.8 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-HighAvailability-debuginfo                14 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-NFV                                       82 kB/s | 2.7 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-NFV-debuginfo                             47 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-RT                                        56 kB/s | 2.7 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-RT-debuginfo                              53 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-ResilientStorage                          31 kB/s | 2.8 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-ResilientStorage-debuginfo                27 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-SAP                                      174 kB/s | 2.7 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-SAP-debuginfo                             23 kB/s | 1.5 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-SAPHANA                                   60 kB/s | 2.7 kB     00:00
Apr 10 06:10:04 rdma06 dnf[738661]: beaker-SAPHANA-debuginfo                         47 kB/s | 1.5 kB     00:00
Apr 10 06:10:05 rdma06 dnf[738661]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 10 06:10:06 rdma06 dnf[738661]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 10 06:10:06 rdma06 dnf[738661]: Metadata cache created.
Apr 10 06:10:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 06:10:06 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 06:10:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.373s CPU time.
Apr 10 06:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:43:30 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr 10 06:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 06:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 06:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 06:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 06:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 06:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:39:45 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 07:39:46 rdma06 dnf[742308]: Updating Subscription Management repositories.
Apr 10 07:39:46 rdma06 dnf[742308]: Unable to read consumer identity
Apr 10 07:39:46 rdma06 dnf[742308]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 07:39:46 rdma06 dnf[742308]: Metadata cache refreshed recently.
Apr 10 07:39:46 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 07:39:46 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 07:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 07:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 07:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 07:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 07:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 07:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 08:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 08:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 08:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 08:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 08:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:17:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 09:17:01 rdma06 dnf[746257]: Updating Subscription Management repositories.
Apr 10 09:17:01 rdma06 dnf[746257]: Unable to read consumer identity
Apr 10 09:17:01 rdma06 dnf[746257]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 09:17:02 rdma06 dnf[746257]: Copr repo for fmf owned by psss                 3.3 kB/s | 3.3 kB     00:01
Apr 10 09:17:02 rdma06 dnf[746257]: beaker-AppStream                                 71 kB/s | 3.2 kB     00:00
Apr 10 09:17:02 rdma06 dnf[746257]: beaker-AppStream-debuginfo                       47 kB/s | 1.5 kB     00:00
Apr 10 09:17:02 rdma06 dnf[746257]: beaker-BaseOS                                   109 kB/s | 2.7 kB     00:00
Apr 10 09:17:02 rdma06 dnf[746257]: beaker-BaseOS-debuginfo                          41 kB/s | 1.5 kB     00:00
Apr 10 09:17:02 rdma06 dnf[746257]: beaker-CRB                                       47 kB/s | 2.7 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-CRB-debuginfo                             77 kB/s | 1.5 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-HighAvailability                          37 kB/s | 2.8 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-HighAvailability-debuginfo                53 kB/s | 1.5 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-NFV                                       86 kB/s | 2.7 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-NFV-debuginfo                             62 kB/s | 1.5 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-RT                                        60 kB/s | 2.7 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-RT-debuginfo                              41 kB/s | 1.5 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-ResilientStorage                          67 kB/s | 2.8 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-ResilientStorage-debuginfo                53 kB/s | 1.5 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-SAP                                      101 kB/s | 2.7 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-SAPHANA                                   67 kB/s | 2.7 kB     00:00
Apr 10 09:17:03 rdma06 dnf[746257]: beaker-SAPHANA-debuginfo                         43 kB/s | 1.5 kB     00:00
Apr 10 09:17:04 rdma06 dnf[746257]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 10 09:17:04 rdma06 dnf[746257]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 10 09:17:05 rdma06 dnf[746257]: Metadata cache created.
Apr 10 09:17:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 09:17:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 09:17:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.433s CPU time.
Apr 10 09:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:49:12 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr 10 09:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 09:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 09:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 09:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 09:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 09:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681136432.8281] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 10 10:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:58:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 10:58:01 rdma06 dnf[750365]: Updating Subscription Management repositories.
Apr 10 10:58:01 rdma06 dnf[750365]: Unable to read consumer identity
Apr 10 10:58:01 rdma06 dnf[750365]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 10:58:01 rdma06 dnf[750365]: Metadata cache refreshed recently.
Apr 10 10:58:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 10:58:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 10:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 10:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 10:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 10:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 10:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 10:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 11:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 11:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 11:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 11:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 11:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:39:45 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 12:39:45 rdma06 dnf[754483]: Updating Subscription Management repositories.
Apr 10 12:39:45 rdma06 dnf[754483]: Unable to read consumer identity
Apr 10 12:39:45 rdma06 dnf[754483]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 12:39:46 rdma06 dnf[754483]: Copr repo for fmf owned by psss                 6.6 kB/s | 3.3 kB     00:00
Apr 10 12:39:46 rdma06 dnf[754483]: beaker-AppStream                                 68 kB/s | 3.2 kB     00:00
Apr 10 12:39:46 rdma06 dnf[754483]: beaker-AppStream-debuginfo                       35 kB/s | 1.5 kB     00:00
Apr 10 12:39:46 rdma06 dnf[754483]: beaker-BaseOS                                    90 kB/s | 2.7 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-BaseOS-debuginfo                          42 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-CRB                                       55 kB/s | 2.7 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-CRB-debuginfo                             80 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-HighAvailability                          79 kB/s | 2.8 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-HighAvailability-debuginfo                39 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-NFV                                      100 kB/s | 2.7 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-NFV-debuginfo                             57 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-RT                                        44 kB/s | 2.7 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-RT-debuginfo                              60 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-ResilientStorage                          68 kB/s | 2.8 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-ResilientStorage-debuginfo                64 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-SAP                                       73 kB/s | 2.7 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-SAP-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-SAPHANA                                   45 kB/s | 2.7 kB     00:00
Apr 10 12:39:47 rdma06 dnf[754483]: beaker-SAPHANA-debuginfo                         45 kB/s | 1.5 kB     00:00
Apr 10 12:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:39:48 rdma06 dnf[754483]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 10 12:39:49 rdma06 dnf[754483]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 10 12:39:49 rdma06 dnf[754483]: Metadata cache created.
Apr 10 12:39:49 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 12:39:49 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 12:39:49 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.447s CPU time.
Apr 10 12:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 12:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 12:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 12:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 12:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 12:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 13:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 13:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 13:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 13:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 13:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:08:18 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 14:08:18 rdma06 dnf[758108]: Updating Subscription Management repositories.
Apr 10 14:08:18 rdma06 dnf[758108]: Unable to read consumer identity
Apr 10 14:08:18 rdma06 dnf[758108]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 14:08:18 rdma06 dnf[758108]: Metadata cache refreshed recently.
Apr 10 14:08:19 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 14:08:19 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 14:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 14:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 14:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 14:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 14:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 14:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:20:18 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 15:20:18 rdma06 dnf[761023]: Updating Subscription Management repositories.
Apr 10 15:20:18 rdma06 dnf[761023]: Unable to read consumer identity
Apr 10 15:20:18 rdma06 dnf[761023]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 15:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:20:18 rdma06 dnf[761023]: Metadata cache refreshed recently.
Apr 10 15:20:19 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 15:20:19 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 15:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 15:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 15:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 15:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 15:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 15:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681158032.8027] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 10 16:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 16:41:01 rdma06 dnf[764291]: Updating Subscription Management repositories.
Apr 10 16:41:01 rdma06 dnf[764291]: Unable to read consumer identity
Apr 10 16:41:01 rdma06 dnf[764291]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 16:41:02 rdma06 dnf[764291]: Copr repo for fmf owned by psss                 2.6 kB/s | 3.3 kB     00:01
Apr 10 16:41:02 rdma06 dnf[764291]: beaker-AppStream                                 58 kB/s | 3.2 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-AppStream-debuginfo                       41 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-BaseOS                                    84 kB/s | 2.7 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-BaseOS-debuginfo                          52 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-CRB                                       57 kB/s | 2.7 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-CRB-debuginfo                             54 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-HighAvailability                          70 kB/s | 2.8 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-HighAvailability-debuginfo                51 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-NFV                                       82 kB/s | 2.7 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-NFV-debuginfo                             65 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-RT                                        62 kB/s | 2.7 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-RT-debuginfo                             106 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-ResilientStorage                          62 kB/s | 2.8 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-ResilientStorage-debuginfo                70 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-SAP                                      154 kB/s | 2.7 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-SAP-debuginfo                             57 kB/s | 1.5 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-SAPHANA                                   68 kB/s | 2.7 kB     00:00
Apr 10 16:41:03 rdma06 dnf[764291]: beaker-SAPHANA-debuginfo                         37 kB/s | 1.5 kB     00:00
Apr 10 16:41:04 rdma06 dnf[764291]: beaker-harness                                  1.5 kB/s | 1.3 kB     00:00
Apr 10 16:41:05 rdma06 dnf[764291]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 10 16:41:05 rdma06 dnf[764291]: Metadata cache created.
Apr 10 16:41:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 16:41:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 16:41:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.437s CPU time.
Apr 10 16:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 16:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 16:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 16:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 16:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 16:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 17:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 17:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 17:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 17:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 17:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:03:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 18:03:49 rdma06 dnf[767677]: Updating Subscription Management repositories.
Apr 10 18:03:49 rdma06 dnf[767677]: Unable to read consumer identity
Apr 10 18:03:49 rdma06 dnf[767677]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 18:03:49 rdma06 dnf[767677]: Metadata cache refreshed recently.
Apr 10 18:03:49 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 18:03:49 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 18:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 18:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 18:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 18:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 18:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 18:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:12:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 19:12:49 rdma06 dnf[770473]: Updating Subscription Management repositories.
Apr 10 19:12:49 rdma06 dnf[770473]: Unable to read consumer identity
Apr 10 19:12:49 rdma06 dnf[770473]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 19:12:49 rdma06 dnf[770473]: Metadata cache refreshed recently.
Apr 10 19:12:49 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 19:12:49 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 19:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 19:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 19:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 19:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 19:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 19:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:51:28 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 10 20:51:28 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 10 20:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:28 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 10 20:51:28 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 10 20:51:28 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 10 20:51:29 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 10 20:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:56:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 20:56:01 rdma06 dnf[774641]: Updating Subscription Management repositories.
Apr 10 20:56:01 rdma06 dnf[774641]: Unable to read consumer identity
Apr 10 20:56:01 rdma06 dnf[774641]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 20:56:02 rdma06 dnf[774641]: Copr repo for fmf owned by psss                 4.0 kB/s | 3.3 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-AppStream                                 50 kB/s | 3.2 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-BaseOS                                    67 kB/s | 2.7 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-BaseOS-debuginfo                          40 kB/s | 1.5 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-CRB                                       47 kB/s | 2.7 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-CRB-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-HighAvailability                          81 kB/s | 2.8 kB     00:00
Apr 10 20:56:02 rdma06 dnf[774641]: beaker-HighAvailability-debuginfo                49 kB/s | 1.5 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-NFV                                       66 kB/s | 2.7 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-NFV-debuginfo                             53 kB/s | 1.5 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-RT                                        72 kB/s | 2.7 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-RT-debuginfo                              80 kB/s | 1.5 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-ResilientStorage                          64 kB/s | 2.8 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-ResilientStorage-debuginfo                37 kB/s | 1.5 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-SAP                                       76 kB/s | 2.7 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-SAP-debuginfo                             56 kB/s | 1.5 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-SAPHANA                                   65 kB/s | 2.7 kB     00:00
Apr 10 20:56:03 rdma06 dnf[774641]: beaker-SAPHANA-debuginfo                         36 kB/s | 1.5 kB     00:00
Apr 10 20:56:04 rdma06 dnf[774641]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 10 20:56:04 rdma06 dnf[774641]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 10 20:56:05 rdma06 dnf[774641]: Metadata cache created.
Apr 10 20:56:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 20:56:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 20:56:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.459s CPU time.
Apr 10 20:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 20:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 20:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 20:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 20:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 20:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 21:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 21:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 21:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 21:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 21:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681179632.8310] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 10 22:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:39:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 10 22:39:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 10 22:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:42:45 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 22:42:46 rdma06 dnf[779028]: Updating Subscription Management repositories.
Apr 10 22:42:46 rdma06 dnf[779028]: Unable to read consumer identity
Apr 10 22:42:46 rdma06 dnf[779028]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 22:42:46 rdma06 dnf[779028]: Metadata cache refreshed recently.
Apr 10 22:42:46 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 22:42:46 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 22:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:43:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 10 22:43:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 10 22:43:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 10 22:43:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 10 22:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 22:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 22:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 22:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 22:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 22:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:48:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 10 23:48:01 rdma06 dnf[781677]: Updating Subscription Management repositories.
Apr 10 23:48:01 rdma06 dnf[781677]: Unable to read consumer identity
Apr 10 23:48:01 rdma06 dnf[781677]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 10 23:48:01 rdma06 dnf[781677]: Metadata cache refreshed recently.
Apr 10 23:48:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 10 23:48:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 10 23:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 10 23:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 10 23:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 10 23:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 10 23:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 10 23:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:00:00 rdma06 systemd[1]: Starting Rotate log files...
Apr 11 00:00:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 11 00:00:00 rdma06 systemd[1]: Finished Rotate log files.
Apr 11 00:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:54:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 00:54:49 rdma06 dnf[784422]: Updating Subscription Management repositories.
Apr 11 00:54:49 rdma06 dnf[784422]: Unable to read consumer identity
Apr 11 00:54:49 rdma06 dnf[784422]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 00:54:50 rdma06 dnf[784422]: Copr repo for fmf owned by psss                 2.9 kB/s | 3.3 kB     00:01
Apr 11 00:54:50 rdma06 dnf[784422]: beaker-AppStream                                 51 kB/s | 3.2 kB     00:00
Apr 11 00:54:50 rdma06 dnf[784422]: beaker-AppStream-debuginfo                       30 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-BaseOS                                    60 kB/s | 2.7 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-BaseOS-debuginfo                          46 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-CRB                                       58 kB/s | 2.7 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-CRB-debuginfo                             72 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-HighAvailability                          91 kB/s | 2.8 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-HighAvailability-debuginfo                63 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-NFV                                       64 kB/s | 2.7 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-NFV-debuginfo                             71 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-RT                                        55 kB/s | 2.7 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-RT-debuginfo                              55 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-ResilientStorage                          72 kB/s | 2.8 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-ResilientStorage-debuginfo                54 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-SAP                                      144 kB/s | 2.7 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-SAPHANA                                  110 kB/s | 2.7 kB     00:00
Apr 11 00:54:51 rdma06 dnf[784422]: beaker-SAPHANA-debuginfo                         35 kB/s | 1.5 kB     00:00
Apr 11 00:54:52 rdma06 dnf[784422]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 11 00:54:53 rdma06 dnf[784422]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 11 00:54:53 rdma06 dnf[784422]: Metadata cache created.
Apr 11 00:54:53 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 00:54:53 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 00:54:53 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.368s CPU time.
Apr 11 00:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 00:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 00:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 00:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 00:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 00:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:01:19 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr 11 01:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 01:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 01:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 01:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 01:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 01:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:33:12 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr 11 02:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 02:54:01 rdma06 dnf[789277]: Updating Subscription Management repositories.
Apr 11 02:54:01 rdma06 dnf[789277]: Unable to read consumer identity
Apr 11 02:54:01 rdma06 dnf[789277]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 02:54:01 rdma06 dnf[789277]: Metadata cache refreshed recently.
Apr 11 02:54:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 02:54:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 02:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 02:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 02:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 02:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 02:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 02:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 03:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 03:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 03:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 03:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 03:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:01:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:01:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:13:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:13:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:15:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:15:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:18:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:18:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:19:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:19:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:19:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:19:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681201232.8378] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 11 04:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:21:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:22:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:22:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:25:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:25:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 04:26:01 rdma06 dnf[793016]: Updating Subscription Management repositories.
Apr 11 04:26:01 rdma06 dnf[793016]: Unable to read consumer identity
Apr 11 04:26:01 rdma06 dnf[793016]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 04:26:03 rdma06 dnf[793016]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-AppStream                                 46 kB/s | 3.2 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-AppStream-debuginfo                       33 kB/s | 1.5 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-BaseOS                                   114 kB/s | 2.7 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-BaseOS-debuginfo                          36 kB/s | 1.5 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-CRB                                       75 kB/s | 2.7 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-CRB-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-HighAvailability                          67 kB/s | 2.8 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-HighAvailability-debuginfo                32 kB/s | 1.5 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-NFV                                       58 kB/s | 2.7 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-NFV-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-RT                                        54 kB/s | 2.7 kB     00:00
Apr 11 04:26:03 rdma06 dnf[793016]: beaker-RT-debuginfo                              77 kB/s | 1.5 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-ResilientStorage                          63 kB/s | 2.8 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-ResilientStorage-debuginfo                67 kB/s | 1.5 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-SAP                                      255 kB/s | 2.7 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-SAP-debuginfo                             59 kB/s | 1.5 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-SAPHANA                                   90 kB/s | 2.7 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-SAPHANA-debuginfo                         32 kB/s | 1.5 kB     00:00
Apr 11 04:26:04 rdma06 dnf[793016]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 11 04:26:05 rdma06 dnf[793016]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 11 04:26:05 rdma06 dnf[793016]: Metadata cache created.
Apr 11 04:26:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 04:26:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 04:26:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.423s CPU time.
Apr 11 04:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:27:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:27:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:29:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:29:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:30:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:30:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:32:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:32:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:32:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:32:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:32:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:32:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:32:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:32:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:33:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:33:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:35:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:35:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:35:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:35:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:36:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:36:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:37:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:37:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:38:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:38:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:39:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:39:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:39:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:39:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:40:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:41:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:41:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:42:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:42:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:42:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:42:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:43:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:43:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:44:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:44:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:44:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:44:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:44:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:44:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:44:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:44:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:44:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:44:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:45:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:45:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:45:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:45:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:45:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:45:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:47:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:47:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:47:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:47:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:48:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:48:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:48:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:48:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:49:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:49:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:49:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:49:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:49:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:49:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:50:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:50:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:50:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:50:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:50:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:50:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:50:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:50:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:51:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:51:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:51:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:51:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:52:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:52:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:52:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:52:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:52:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:52:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:53:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:53:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:53:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:53:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:53:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:53:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:54:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:54:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:54:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:54:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:55:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:55:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:55:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:55:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:55:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:55:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:55:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:55:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:55:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:55:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:56:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:56:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:56:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:56:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:57:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:57:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:58:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:58:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:58:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:58:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:58:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:58:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:58:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:58:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:58:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:58:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:58:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:58:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:59:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:59:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:59:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:59:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:59:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:59:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:59:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:59:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 04:59:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:59:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 04:59:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 04:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 04:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 04:59:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:00:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:00:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:00:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:00:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:00:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:00:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:00:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:00:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:01:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:01:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:01:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:01:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:01:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:01:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:01:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:01:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:02:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:02:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:02:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:02:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:02:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:02:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:02:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:02:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:02:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:02:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:03:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:03:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:03:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:03:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:03:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:03:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:04:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:04:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:04:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:04:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:04:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:04:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:04:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:04:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:04:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:04:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:04:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:05:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:05:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:05:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:05:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:05:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:05:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:05:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:05:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:05:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:05:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:05:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:05:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:06:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:06:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:06:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:06:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:06:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:06:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:06:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:06:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:06:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:06:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:06:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:06:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:07:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:07:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:07:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:07:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:07:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:07:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:07:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:07:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:07:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:08:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:08:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:08:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:08:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:08:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:08:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:08:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:08:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:08:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:08:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:08:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:08:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:09:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:09:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:09:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:09:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:09:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:09:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:09:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:09:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:09:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:09:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:09:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:09:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:10:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:10:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:10:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:10:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:10:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:10:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:10:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:10:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:10:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:10:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:10:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:10:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:11:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:11:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:11:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:11:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:11:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:11:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:11:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:11:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:11:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:11:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:12:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:12:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:12:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:12:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:12:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:12:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:12:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:12:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:12:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:13:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:13:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:13:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:13:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:13:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:13:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:13:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:13:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:13:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:13:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:14:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:14:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:14:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:14:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:14:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:14:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:14:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:14:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:14:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:14:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:14:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:14:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:15:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:15:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:15:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:15:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:15:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:15:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:15:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:15:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:15:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:15:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:16:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:16:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:16:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:16:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:16:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:16:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:16:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:16:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:16:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:16:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:16:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:16:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:17:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:17:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:17:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:17:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:17:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:17:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:17:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:17:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:17:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:17:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:17:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:17:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:18:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:18:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:18:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:18:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:18:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:18:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:18:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:18:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:18:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:18:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:19:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:19:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:19:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:19:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:19:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:19:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:19:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:19:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:20:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:20:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:20:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:20:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:20:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:20:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:20:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:20:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:20:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:20:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:20:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:20:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:21:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:21:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:21:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:21:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:21:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:21:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:21:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:21:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:21:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:21:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:22:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:22:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:22:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:22:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:22:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:22:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:22:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:22:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:22:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:22:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:23:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:23:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:23:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:23:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:23:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:23:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:23:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:23:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:23:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:23:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:23:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:23:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:24:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:24:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:24:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:24:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:24:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:24:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:24:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:24:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:24:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:24:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:24:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:24:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:25:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:25:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:25:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:25:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:25:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:25:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:25:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:25:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:25:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:25:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:26:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:26:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:26:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:26:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:26:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:26:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:26:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:26:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:26:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:26:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:26:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:26:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:27:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 05:27:01 rdma06 dnf[795521]: Updating Subscription Management repositories.
Apr 11 05:27:01 rdma06 dnf[795521]: Unable to read consumer identity
Apr 11 05:27:01 rdma06 dnf[795521]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 05:27:01 rdma06 dnf[795521]: Metadata cache refreshed recently.
Apr 11 05:27:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 05:27:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 05:27:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:27:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:27:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:27:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:27:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:27:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:27:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:27:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:27:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:27:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:28:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:28:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:28:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:28:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:28:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:28:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:28:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:28:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:28:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:29:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:29:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:29:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:29:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:29:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:29:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:29:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:29:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:29:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:29:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:30:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:30:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:30:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:30:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:30:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:30:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:30:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:30:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:30:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:30:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:31:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:31:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:31:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:31:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:31:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:31:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:31:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:31:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:31:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:31:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:31:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:31:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:32:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:32:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:32:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:32:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:33:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:33:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:33:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:33:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:33:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:33:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:33:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:33:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:33:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:33:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:34:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:34:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:34:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:34:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:34:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:34:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:34:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:34:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:34:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:34:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:34:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:34:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:35:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:35:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:35:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:35:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:35:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:35:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:35:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:35:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:36:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:36:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:36:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:36:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:36:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:36:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:36:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:36:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:36:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:36:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:37:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:37:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:37:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:37:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:37:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:37:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:37:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:37:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:37:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:37:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:38:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:38:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:38:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:38:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:38:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:38:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:38:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:38:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:38:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:38:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:39:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:39:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:39:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:39:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:39:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:39:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:39:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:39:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:40:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:40:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:40:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:40:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:40:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:40:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:40:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:40:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:40:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:40:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:41:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:41:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:41:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:41:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:41:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:41:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:41:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:41:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:41:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:41:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:42:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:42:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:42:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:42:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:42:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:42:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:42:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:42:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:43:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:43:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:43:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:43:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:43:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:43:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:43:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:43:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:43:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:43:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:44:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:44:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:45:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:45:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:45:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:45:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:45:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:45:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:46:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:46:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:46:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:46:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:46:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:46:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:46:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:46:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:46:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:46:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:46:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:46:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:47:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:47:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:47:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:47:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:47:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:47:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:48:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:48:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:48:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:48:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:48:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:48:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:48:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:48:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:49:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:49:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:49:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:49:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:49:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:49:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:50:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:50:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:50:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:50:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:51:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:51:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:51:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:51:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:51:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:51:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:51:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:51:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:52:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:52:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:52:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:52:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:52:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:52:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:53:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:53:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:53:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:53:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:53:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:53:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:54:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:54:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:54:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:54:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:54:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:54:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:54:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:54:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:55:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:55:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:56:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:56:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:56:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:56:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:56:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:56:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:57:08 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:57:08 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:57:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:57:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:57:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:57:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:57:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:57:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:57:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:57:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 05:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 05:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 05:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 05:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 05:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:00:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:00:28 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:00:58 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:00:58 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:01:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:01:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:02:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:02:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:03:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:38 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:03:38 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:03:48 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:03:48 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:04:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:07:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:07:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:07:28 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:11:18 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:11:18 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 06:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 06:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 06:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 06:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 06:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:05:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 07:05:01 rdma06 dnf[799496]: Updating Subscription Management repositories.
Apr 11 07:05:01 rdma06 dnf[799496]: Unable to read consumer identity
Apr 11 07:05:01 rdma06 dnf[799496]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 07:05:01 rdma06 dnf[799496]: Metadata cache refreshed recently.
Apr 11 07:05:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 07:05:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 07:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 07:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 07:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 07:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 07:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 07:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:45:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 08:45:01 rdma06 dnf[803545]: Updating Subscription Management repositories.
Apr 11 08:45:01 rdma06 dnf[803545]: Unable to read consumer identity
Apr 11 08:45:01 rdma06 dnf[803545]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 08:45:02 rdma06 dnf[803545]: Copr repo for fmf owned by psss                 3.6 kB/s | 3.3 kB     00:00
Apr 11 08:45:02 rdma06 dnf[803545]: beaker-AppStream                                 16 kB/s | 3.2 kB     00:00
Apr 11 08:45:02 rdma06 dnf[803545]: beaker-AppStream-debuginfo                      8.8 kB/s | 1.5 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-BaseOS                                    19 kB/s | 2.7 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-BaseOS-debuginfo                          14 kB/s | 1.5 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-CRB                                       25 kB/s | 2.7 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-CRB-debuginfo                             68 kB/s | 1.5 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-HighAvailability                          47 kB/s | 2.8 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-HighAvailability-debuginfo                43 kB/s | 1.5 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-NFV                                       33 kB/s | 2.7 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-NFV-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-RT                                        41 kB/s | 2.7 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-RT-debuginfo                              64 kB/s | 1.5 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-ResilientStorage                          64 kB/s | 2.8 kB     00:00
Apr 11 08:45:03 rdma06 dnf[803545]: beaker-ResilientStorage-debuginfo                55 kB/s | 1.5 kB     00:00
Apr 11 08:45:04 rdma06 dnf[803545]: beaker-SAP                                       72 kB/s | 2.7 kB     00:00
Apr 11 08:45:04 rdma06 dnf[803545]: beaker-SAP-debuginfo                             41 kB/s | 1.5 kB     00:00
Apr 11 08:45:04 rdma06 dnf[803545]: beaker-SAPHANA                                   58 kB/s | 2.7 kB     00:00
Apr 11 08:45:04 rdma06 dnf[803545]: beaker-SAPHANA-debuginfo                         35 kB/s | 1.5 kB     00:00
Apr 11 08:45:04 rdma06 dnf[803545]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 11 08:45:05 rdma06 dnf[803545]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 11 08:45:05 rdma06 dnf[803545]: Metadata cache created.
Apr 11 08:45:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 08:45:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 08:45:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.396s CPU time.
Apr 11 08:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:47:41 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr 11 08:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 08:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 08:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 08:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 08:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 08:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 09:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 09:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 09:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 09:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 09:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:16:40 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr 11 10:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681222832.9524] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 11 10:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:24:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 10:24:01 rdma06 dnf[807589]: Updating Subscription Management repositories.
Apr 11 10:24:01 rdma06 dnf[807589]: Unable to read consumer identity
Apr 11 10:24:01 rdma06 dnf[807589]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 10:24:01 rdma06 dnf[807589]: Metadata cache refreshed recently.
Apr 11 10:24:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 10:24:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 10:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 10:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 10:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 10:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 10:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 10:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 11:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 11:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 11:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 11:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 11:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:24:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 12:24:01 rdma06 dnf[812451]: Updating Subscription Management repositories.
Apr 11 12:24:01 rdma06 dnf[812451]: Unable to read consumer identity
Apr 11 12:24:01 rdma06 dnf[812451]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 12:24:03 rdma06 dnf[812451]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-AppStream                                 37 kB/s | 3.2 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-BaseOS                                    79 kB/s | 2.7 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-BaseOS-debuginfo                          31 kB/s | 1.5 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-CRB                                       58 kB/s | 2.7 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-CRB-debuginfo                             76 kB/s | 1.5 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-HighAvailability                          79 kB/s | 2.8 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-HighAvailability-debuginfo                42 kB/s | 1.5 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-NFV                                       66 kB/s | 2.7 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-NFV-debuginfo                             65 kB/s | 1.5 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-RT                                        62 kB/s | 2.7 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-RT-debuginfo                              71 kB/s | 1.5 kB     00:00
Apr 11 12:24:03 rdma06 dnf[812451]: beaker-ResilientStorage                          81 kB/s | 2.8 kB     00:00
Apr 11 12:24:04 rdma06 dnf[812451]: beaker-ResilientStorage-debuginfo                51 kB/s | 1.5 kB     00:00
Apr 11 12:24:04 rdma06 dnf[812451]: beaker-SAP                                      166 kB/s | 2.7 kB     00:00
Apr 11 12:24:04 rdma06 dnf[812451]: beaker-SAP-debuginfo                             75 kB/s | 1.5 kB     00:00
Apr 11 12:24:04 rdma06 dnf[812451]: beaker-SAPHANA                                   75 kB/s | 2.7 kB     00:00
Apr 11 12:24:04 rdma06 dnf[812451]: beaker-SAPHANA-debuginfo                         46 kB/s | 1.5 kB     00:00
Apr 11 12:24:04 rdma06 dnf[812451]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 11 12:24:05 rdma06 dnf[812451]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 11 12:24:05 rdma06 dnf[812451]: Metadata cache created.
Apr 11 12:24:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 12:24:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 12:24:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.439s CPU time.
Apr 11 12:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 12:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 12:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 12:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 12:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 12:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 13:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 13:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 13:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 13:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 13:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 14:08:01 rdma06 dnf[816685]: Updating Subscription Management repositories.
Apr 11 14:08:01 rdma06 dnf[816685]: Unable to read consumer identity
Apr 11 14:08:01 rdma06 dnf[816685]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 14:08:01 rdma06 dnf[816685]: Metadata cache refreshed recently.
Apr 11 14:08:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 14:08:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 14:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 14:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 14:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 14:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 14:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 14:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:21:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 15:21:01 rdma06 dnf[819639]: Updating Subscription Management repositories.
Apr 11 15:21:01 rdma06 dnf[819639]: Unable to read consumer identity
Apr 11 15:21:01 rdma06 dnf[819639]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 15:21:01 rdma06 dnf[819639]: Metadata cache refreshed recently.
Apr 11 15:21:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 15:21:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 15:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 15:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 15:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 15:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 15:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 15:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681244432.8659] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 11 16:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:36:53 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 16:36:54 rdma06 dnf[822714]: Updating Subscription Management repositories.
Apr 11 16:36:54 rdma06 dnf[822714]: Unable to read consumer identity
Apr 11 16:36:54 rdma06 dnf[822714]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 16:36:55 rdma06 dnf[822714]: Copr repo for fmf owned by psss                 3.5 kB/s | 3.3 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-AppStream                                 33 kB/s | 3.2 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-AppStream-debuginfo                       47 kB/s | 1.5 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-BaseOS                                    84 kB/s | 2.7 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-BaseOS-debuginfo                          38 kB/s | 1.5 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-CRB                                       50 kB/s | 2.7 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-CRB-debuginfo                             46 kB/s | 1.5 kB     00:00
Apr 11 16:36:55 rdma06 dnf[822714]: beaker-HighAvailability                          66 kB/s | 2.8 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-HighAvailability-debuginfo                49 kB/s | 1.5 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-NFV                                       61 kB/s | 2.7 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-NFV-debuginfo                             38 kB/s | 1.5 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-RT                                        58 kB/s | 2.7 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-RT-debuginfo                              44 kB/s | 1.5 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-ResilientStorage                          63 kB/s | 2.8 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-ResilientStorage-debuginfo                53 kB/s | 1.5 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-SAP                                      153 kB/s | 2.7 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-SAP-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-SAPHANA                                   80 kB/s | 2.7 kB     00:00
Apr 11 16:36:56 rdma06 dnf[822714]: beaker-SAPHANA-debuginfo                         40 kB/s | 1.5 kB     00:00
Apr 11 16:36:57 rdma06 dnf[822714]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 11 16:36:57 rdma06 dnf[822714]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 11 16:36:58 rdma06 dnf[822714]: Metadata cache created.
Apr 11 16:36:58 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 16:36:58 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 16:36:58 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.402s CPU time.
Apr 11 16:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 16:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 16:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 16:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 16:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 16:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 17:54:01 rdma06 dnf[825857]: Updating Subscription Management repositories.
Apr 11 17:54:01 rdma06 dnf[825857]: Unable to read consumer identity
Apr 11 17:54:01 rdma06 dnf[825857]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 17:54:01 rdma06 dnf[825857]: Metadata cache refreshed recently.
Apr 11 17:54:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 17:54:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 17:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 17:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 17:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 17:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 17:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 17:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 18:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 18:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 18:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 18:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 18:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:49 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 19:27:50 rdma06 dnf[829672]: Updating Subscription Management repositories.
Apr 11 19:27:50 rdma06 dnf[829672]: Unable to read consumer identity
Apr 11 19:27:50 rdma06 dnf[829672]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 19:27:50 rdma06 dnf[829672]: Metadata cache refreshed recently.
Apr 11 19:27:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 19:27:50 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 19:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 19:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 19:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 19:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 19:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 19:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:51:49 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 20:51:50 rdma06 dnf[833071]: Updating Subscription Management repositories.
Apr 11 20:51:50 rdma06 dnf[833071]: Unable to read consumer identity
Apr 11 20:51:50 rdma06 dnf[833071]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 20:51:51 rdma06 dnf[833071]: Copr repo for fmf owned by psss                 2.8 kB/s | 3.3 kB     00:01
Apr 11 20:51:51 rdma06 dnf[833071]: beaker-AppStream                                 50 kB/s | 3.2 kB     00:00
Apr 11 20:51:51 rdma06 dnf[833071]: beaker-AppStream-debuginfo                       36 kB/s | 1.5 kB     00:00
Apr 11 20:51:51 rdma06 dnf[833071]: beaker-BaseOS                                    52 kB/s | 2.7 kB     00:00
Apr 11 20:51:51 rdma06 dnf[833071]: beaker-BaseOS-debuginfo                          35 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-CRB                                      7.6 kB/s | 2.7 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-CRB-debuginfo                             60 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-HighAvailability                          70 kB/s | 2.8 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-HighAvailability-debuginfo                41 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-NFV                                      100 kB/s | 2.7 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-NFV-debuginfo                             43 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-RT                                        54 kB/s | 2.7 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-RT-debuginfo                              43 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-ResilientStorage                          62 kB/s | 2.8 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-ResilientStorage-debuginfo                42 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-SAP                                       84 kB/s | 2.7 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-SAP-debuginfo                             48 kB/s | 1.5 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-SAPHANA                                   47 kB/s | 2.7 kB     00:00
Apr 11 20:51:52 rdma06 dnf[833071]: beaker-SAPHANA-debuginfo                         29 kB/s | 1.5 kB     00:00
Apr 11 20:51:53 rdma06 dnf[833071]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 11 20:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:51:54 rdma06 dnf[833071]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 11 20:51:54 rdma06 dnf[833071]: Metadata cache created.
Apr 11 20:51:54 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 20:51:54 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 20:51:54 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.429s CPU time.
Apr 11 20:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 20:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 20:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 20:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 20:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 20:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 21:53:01 rdma06 dnf[835570]: Updating Subscription Management repositories.
Apr 11 21:53:01 rdma06 dnf[835570]: Unable to read consumer identity
Apr 11 21:53:01 rdma06 dnf[835570]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 21:53:01 rdma06 dnf[835570]: Metadata cache refreshed recently.
Apr 11 21:53:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 21:53:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 21:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 21:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 21:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 21:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 21:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 21:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681266032.9313] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 11 22:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:40:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 11 22:40:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 11 22:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:44:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 11 22:44:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 11 22:44:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 11 22:44:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 11 22:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 22:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 22:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 22:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 22:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 22:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:17:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 11 23:17:01 rdma06 dnf[838987]: Updating Subscription Management repositories.
Apr 11 23:17:01 rdma06 dnf[838987]: Unable to read consumer identity
Apr 11 23:17:01 rdma06 dnf[838987]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 11 23:17:01 rdma06 dnf[838987]: Metadata cache refreshed recently.
Apr 11 23:17:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 11 23:17:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 11 23:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 11 23:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 11 23:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 11 23:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 11 23:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 11 23:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:00:49 rdma06 systemd[1]: Starting Rotate log files...
Apr 12 00:00:49 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 12 00:00:49 rdma06 systemd[1]: Finished Rotate log files.
Apr 12 00:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:37:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 00:37:01 rdma06 dnf[842243]: Updating Subscription Management repositories.
Apr 12 00:37:01 rdma06 dnf[842243]: Unable to read consumer identity
Apr 12 00:37:01 rdma06 dnf[842243]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 00:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:37:02 rdma06 dnf[842243]: Copr repo for fmf owned by psss                 2.8 kB/s | 3.3 kB     00:01
Apr 12 00:37:02 rdma06 dnf[842243]: beaker-AppStream                                 36 kB/s | 3.2 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-AppStream-debuginfo                      9.1 kB/s | 1.5 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-BaseOS                                    94 kB/s | 2.7 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-BaseOS-debuginfo                          11 kB/s | 1.5 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-CRB                                       32 kB/s | 2.7 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-CRB-debuginfo                             19 kB/s | 1.5 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-HighAvailability                          64 kB/s | 2.8 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-HighAvailability-debuginfo                26 kB/s | 1.5 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-NFV                                       29 kB/s | 2.7 kB     00:00
Apr 12 00:37:03 rdma06 dnf[842243]: beaker-NFV-debuginfo                             33 kB/s | 1.5 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-RT                                        26 kB/s | 2.7 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-RT-debuginfo                              26 kB/s | 1.5 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-ResilientStorage                          25 kB/s | 2.8 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-ResilientStorage-debuginfo                53 kB/s | 1.5 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-SAP                                       43 kB/s | 2.7 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-SAP-debuginfo                             35 kB/s | 1.5 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-SAPHANA                                   23 kB/s | 2.7 kB     00:00
Apr 12 00:37:04 rdma06 dnf[842243]: beaker-SAPHANA-debuginfo                         39 kB/s | 1.5 kB     00:00
Apr 12 00:37:05 rdma06 dnf[842243]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 12 00:37:05 rdma06 dnf[842243]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 12 00:37:06 rdma06 dnf[842243]: Metadata cache created.
Apr 12 00:37:06 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 00:37:06 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 00:37:06 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.471s CPU time.
Apr 12 00:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 00:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 00:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 00:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 00:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 00:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:46:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 01:46:01 rdma06 dnf[845076]: Updating Subscription Management repositories.
Apr 12 01:46:01 rdma06 dnf[845076]: Unable to read consumer identity
Apr 12 01:46:01 rdma06 dnf[845076]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 01:46:01 rdma06 dnf[845076]: Metadata cache refreshed recently.
Apr 12 01:46:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 01:46:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 01:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 01:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 01:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 01:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 01:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 01:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 02:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 02:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 02:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 02:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 02:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:11:15 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 03:11:16 rdma06 dnf[848543]: Updating Subscription Management repositories.
Apr 12 03:11:16 rdma06 dnf[848543]: Unable to read consumer identity
Apr 12 03:11:16 rdma06 dnf[848543]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 03:11:16 rdma06 dnf[848543]: Metadata cache refreshed recently.
Apr 12 03:11:16 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 03:11:16 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 03:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 03:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 03:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 03:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 03:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 03:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:12:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 04:12:01 rdma06 dnf[851010]: Updating Subscription Management repositories.
Apr 12 04:12:01 rdma06 dnf[851010]: Unable to read consumer identity
Apr 12 04:12:01 rdma06 dnf[851010]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 04:12:02 rdma06 dnf[851010]: Copr repo for fmf owned by psss                 4.1 kB/s | 3.3 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-AppStream                                 26 kB/s | 3.2 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-AppStream-debuginfo                       50 kB/s | 1.5 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-BaseOS                                   112 kB/s | 2.7 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-BaseOS-debuginfo                          46 kB/s | 1.5 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-CRB                                       67 kB/s | 2.7 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-CRB-debuginfo                             90 kB/s | 1.5 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-HighAvailability                          77 kB/s | 2.8 kB     00:00
Apr 12 04:12:02 rdma06 dnf[851010]: beaker-HighAvailability-debuginfo                31 kB/s | 1.5 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-NFV                                       76 kB/s | 2.7 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-NFV-debuginfo                             46 kB/s | 1.5 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-RT                                        58 kB/s | 2.7 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-RT-debuginfo                              35 kB/s | 1.5 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-ResilientStorage                          53 kB/s | 2.8 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-ResilientStorage-debuginfo                56 kB/s | 1.5 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-SAP                                       79 kB/s | 2.7 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-SAP-debuginfo                             41 kB/s | 1.5 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-SAPHANA                                   60 kB/s | 2.7 kB     00:00
Apr 12 04:12:03 rdma06 dnf[851010]: beaker-SAPHANA-debuginfo                         28 kB/s | 1.5 kB     00:00
Apr 12 04:12:04 rdma06 dnf[851010]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 12 04:12:04 rdma06 dnf[851010]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 12 04:12:05 rdma06 dnf[851010]: Metadata cache created.
Apr 12 04:12:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 04:12:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 04:12:05 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.417s CPU time.
Apr 12 04:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681287632.8953] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 12 04:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 04:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 04:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 04:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 04:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 04:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 05:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 05:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 05:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 05:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 05:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:09:49 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 06:09:50 rdma06 dnf[855804]: Updating Subscription Management repositories.
Apr 12 06:09:50 rdma06 dnf[855804]: Unable to read consumer identity
Apr 12 06:09:50 rdma06 dnf[855804]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 06:09:50 rdma06 dnf[855804]: Metadata cache refreshed recently.
Apr 12 06:09:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 06:09:50 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 06:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 06:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 06:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 06:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 06:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 06:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:09:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:09:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:10:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:10:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:10:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:10:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:11:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:11:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:15:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:15:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:16:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:16:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:17:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:17:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:18:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:18:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:20:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:20:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:20:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:20:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:20:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:20:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:21:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:21:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:22:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:22:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:22:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:22:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:22:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:22:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:23:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:23:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:23:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:23:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:27:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:27:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:27:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:27:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:28:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:28:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:28:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:28:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:28:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:28:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:29:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:29:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:29:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:29:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:29:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:29:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:30:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 07:30:49 rdma06 dnf[859078]: Updating Subscription Management repositories.
Apr 12 07:30:49 rdma06 dnf[859078]: Unable to read consumer identity
Apr 12 07:30:49 rdma06 dnf[859078]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 07:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:30:50 rdma06 dnf[859078]: Copr repo for fmf owned by psss                 2.3 kB/s | 3.3 kB     00:01
Apr 12 07:30:50 rdma06 dnf[859078]: beaker-AppStream                                 70 kB/s | 3.2 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-AppStream-debuginfo                       33 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-BaseOS                                   144 kB/s | 2.7 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-BaseOS-debuginfo                          50 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-CRB                                       53 kB/s | 2.7 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-CRB-debuginfo                             47 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-HighAvailability                          43 kB/s | 2.8 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-HighAvailability-debuginfo                22 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-NFV                                       35 kB/s | 2.7 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-NFV-debuginfo                             19 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-RT                                        30 kB/s | 2.7 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-RT-debuginfo                              26 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-ResilientStorage                          80 kB/s | 2.8 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-ResilientStorage-debuginfo                41 kB/s | 1.5 kB     00:00
Apr 12 07:30:51 rdma06 dnf[859078]: beaker-SAP                                       93 kB/s | 2.7 kB     00:00
Apr 12 07:30:52 rdma06 dnf[859078]: beaker-SAP-debuginfo                             62 kB/s | 1.5 kB     00:00
Apr 12 07:30:52 rdma06 dnf[859078]: beaker-SAPHANA                                   66 kB/s | 2.7 kB     00:00
Apr 12 07:30:52 rdma06 dnf[859078]: beaker-SAPHANA-debuginfo                         57 kB/s | 1.5 kB     00:00
Apr 12 07:30:52 rdma06 dnf[859078]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 12 07:30:53 rdma06 dnf[859078]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 12 07:30:53 rdma06 dnf[859078]: Metadata cache created.
Apr 12 07:30:53 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 07:30:53 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 07:30:53 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.421s CPU time.
Apr 12 07:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:31:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:31:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:31:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:31:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:32:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:32:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:33:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:33:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:34:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:34:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:34:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:34:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:35:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:35:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:35:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:35:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:35:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:35:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:35:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:35:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:37:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:37:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:37:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:37:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:37:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:37:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:37:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:37:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:38:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:38:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:38:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:38:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:39:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:39:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:39:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:39:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:40:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:40:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:40:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:40:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:40:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:40:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:40:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:40:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:40:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:40:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:41:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:41:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:41:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:41:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:41:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:41:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:42:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:42:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:43:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:43:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:43:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:43:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:43:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:43:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:44:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:44:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:44:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:44:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:45:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:45:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:45:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:45:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:46:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:46:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:46:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:46:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:47:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:47:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:47:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:47:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:47:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:47:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:47:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:47:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:47:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:47:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:47:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:47:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:48:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:48:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:48:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:48:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:48:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:48:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:49:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:49:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:49:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:49:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:49:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:50:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:50:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:50:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:50:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:50:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:50:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:50:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:50:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:51:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:51:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:51:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:51:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:51:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:51:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:51:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:51:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:51:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:51:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:52:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:52:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:52:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:52:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:53:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:53:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:53:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:53:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:53:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:53:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:53:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:53:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:54:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:54:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:54:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:54:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:54:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:54:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:54:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:54:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:55:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:55:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:55:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:55:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:55:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:55:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:55:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:56:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:56:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:56:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:56:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:56:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:56:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:56:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:56:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:56:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:56:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:57:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:57:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:57:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:57:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:57:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:57:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:57:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:57:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:57:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:57:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:57:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:57:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:58:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:58:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:58:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:58:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:58:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:58:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:58:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:58:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:58:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:58:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:58:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:58:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:59:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:59:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:59:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:59:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:59:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:59:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:59:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:59:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 07:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 07:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 07:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 07:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 07:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:00:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:00:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:00:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:00:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:00:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:00:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:00:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:00:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:00:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:00:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:00:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:00:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:01:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:01:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:01:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:01:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:01:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:01:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:01:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:01:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:01:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:01:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:01:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:01:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:02:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:02:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:02:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:02:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:02:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:02:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:02:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:02:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:02:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:02:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:02:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:02:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:03:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:03:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:03:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:03:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:03:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:03:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:03:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:03:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:03:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:03:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:04:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:04:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:04:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:04:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:04:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:04:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:04:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:04:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:05:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:05:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:05:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:05:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:05:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:05:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:06:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:06:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:06:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:06:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:06:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:06:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:06:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:06:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:06:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:06:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:07:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:07:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:07:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:07:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:07:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:07:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:07:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:07:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:07:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:07:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:08:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:08:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:08:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:08:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:08:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:08:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:08:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:08:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:08:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:08:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:08:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:08:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:09:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:09:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:09:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:09:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:09:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:09:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:09:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:09:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:09:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:09:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:10:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:10:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:10:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:10:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:10:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:10:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:10:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:10:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:11:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:11:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:11:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:11:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:11:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:11:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:11:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:11:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:11:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:11:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:12:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:12:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:12:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:12:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:12:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:12:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:12:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:12:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:12:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:12:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:12:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:12:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:13:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:13:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:13:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:13:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:13:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:13:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:13:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:13:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:13:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:13:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:13:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:13:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:14:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:14:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:14:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:14:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:14:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:14:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:14:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:14:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:14:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:14:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:14:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:14:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:15:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:15:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:15:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:15:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:15:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:15:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:15:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:15:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:15:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:15:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:16:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:16:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:16:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:16:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:16:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:16:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:16:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:16:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:16:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:16:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:17:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:17:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:17:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:17:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:17:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:17:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:18:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:18:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:18:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:18:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:18:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:18:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:18:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:18:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:18:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:18:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:19:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:19:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:19:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:19:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:19:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:19:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:19:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:19:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:19:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:19:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:19:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:19:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:20:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:20:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:20:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:20:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:20:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:20:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:21:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:21:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:21:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:21:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:21:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:21:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:21:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:21:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:21:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:21:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:22:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:22:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:22:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:22:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:22:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:22:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:23:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:23:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:23:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:23:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:23:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:23:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:23:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:23:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:24:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:24:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:24:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:24:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:24:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:24:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:24:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:25:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:25:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:25:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:25:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:25:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:25:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:25:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:25:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:25:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:25:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:25:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:25:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:26:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:26:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:26:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:26:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:26:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:26:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:26:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:26:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:26:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:26:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:26:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:26:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:27:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:27:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:27:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:27:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:27:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:27:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:27:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:27:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:28:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:28:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:28:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:28:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:28:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:28:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:29:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:29:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:29:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:29:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:29:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:29:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:30:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:30:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:30:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:30:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:30:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:30:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:30:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:31:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:31:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:31:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:31:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:31:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:31:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:31:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:31:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:32:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 08:32:01 rdma06 dnf[861583]: Updating Subscription Management repositories.
Apr 12 08:32:01 rdma06 dnf[861583]: Unable to read consumer identity
Apr 12 08:32:01 rdma06 dnf[861583]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 08:32:01 rdma06 dnf[861583]: Metadata cache refreshed recently.
Apr 12 08:32:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 08:32:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 08:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:32:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:32:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:32:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:32:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:32:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:32:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:32:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:32:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:32:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:32:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:33:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:33:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:33:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:33:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:33:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:33:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:33:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:33:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:33:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:33:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:34:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:34:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:34:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:34:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:34:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:35:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:35:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:35:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:35:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:36:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:36:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:36:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:36:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:36:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:37:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:37:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:37:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:37:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:38:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:38:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:38:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:38:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:38:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:38:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:38:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:38:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:39:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:39:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:39:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:39:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:39:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:39:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:39:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:39:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:40:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:40:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:41:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:41:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:41:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:41:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:42:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:42:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:42:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:42:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:42:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:42:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:42:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:42:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:42:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:42:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:43:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:43:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:43:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:43:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:43:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:43:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:44:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:44:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:44:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:44:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:44:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:44:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:44:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:44:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:45:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:45:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:45:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:46:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:46:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:46:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:46:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:46:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:46:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:46:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:46:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:48:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:48:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:48:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:49:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:49:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:49:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:49:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:49:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:49:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:49:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:50:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:50:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:50:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:50:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:51:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:52:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:52:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:52:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:52:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:52:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:53:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:53:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:53:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:53:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:54:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:54:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:54:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:54:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:55:19 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:55:19 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:55:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:55:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:56:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:56:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:59:49 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:49 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 08:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 08:59:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 08:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 08:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 08:59:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:03:29 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:03:29 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:04:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:04:59 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:04:59 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:05:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:05:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:06:09 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:06:09 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:39 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:07:39 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 09:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 09:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 09:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 09:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 09:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:15:50 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 10:15:50 rdma06 dnf[865799]: Updating Subscription Management repositories.
Apr 12 10:15:50 rdma06 dnf[865799]: Unable to read consumer identity
Apr 12 10:15:50 rdma06 dnf[865799]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 10:15:50 rdma06 dnf[865799]: Metadata cache refreshed recently.
Apr 12 10:15:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 10:15:50 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 10:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681309232.8758] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 12 10:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 10:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 10:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 10:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 10:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 10:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:06:39 rdma06 systemd[1]: session-11.scope: Deactivated successfully.
Apr 12 11:06:39 rdma06 systemd[1]: session-11.scope: Consumed 8.598s CPU time.
Apr 12 11:06:39 rdma06 systemd-logind[1064]: Session 11 logged out. Waiting for processes to exit.
Apr 12 11:06:39 rdma06 systemd-logind[1064]: Removed session 11.
Apr 12 11:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 11:20:01 rdma06 dnf[868406]: Updating Subscription Management repositories.
Apr 12 11:20:01 rdma06 dnf[868406]: Unable to read consumer identity
Apr 12 11:20:01 rdma06 dnf[868406]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 11:20:02 rdma06 dnf[868406]: Copr repo for fmf owned by psss                 5.5 kB/s | 3.3 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-AppStream                                 61 kB/s | 3.2 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-AppStream-debuginfo                       18 kB/s | 1.5 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-BaseOS                                   122 kB/s | 2.7 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-BaseOS-debuginfo                          53 kB/s | 1.5 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-CRB                                       75 kB/s | 2.7 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-CRB-debuginfo                             71 kB/s | 1.5 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-HighAvailability                          76 kB/s | 2.8 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-HighAvailability-debuginfo                42 kB/s | 1.5 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-NFV                                       74 kB/s | 2.7 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-NFV-debuginfo                             42 kB/s | 1.5 kB     00:00
Apr 12 11:20:02 rdma06 dnf[868406]: beaker-RT                                        57 kB/s | 2.7 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-RT-debuginfo                              35 kB/s | 1.5 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-ResilientStorage                          75 kB/s | 2.8 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-ResilientStorage-debuginfo                74 kB/s | 1.5 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-SAP                                       96 kB/s | 2.7 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-SAP-debuginfo                             58 kB/s | 1.5 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-SAPHANA                                   63 kB/s | 2.7 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-SAPHANA-debuginfo                         38 kB/s | 1.5 kB     00:00
Apr 12 11:20:03 rdma06 dnf[868406]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 12 11:20:04 rdma06 dnf[868406]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 12 11:20:04 rdma06 dnf[868406]: Metadata cache created.
Apr 12 11:20:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 11:20:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 11:20:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.466s CPU time.
Apr 12 11:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 11:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 11:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 11:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 11:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 11:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:21:44 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 12:21:45 rdma06 dnf[870924]: Updating Subscription Management repositories.
Apr 12 12:21:45 rdma06 dnf[870924]: Unable to read consumer identity
Apr 12 12:21:45 rdma06 dnf[870924]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 12:21:45 rdma06 dnf[870924]: Metadata cache refreshed recently.
Apr 12 12:21:45 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 12:21:45 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 12:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 12:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 12:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 12:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 12:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 12:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 13:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 13:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 13:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 13:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 13:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:02:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 14:02:01 rdma06 dnf[874992]: Updating Subscription Management repositories.
Apr 12 14:02:01 rdma06 dnf[874992]: Unable to read consumer identity
Apr 12 14:02:01 rdma06 dnf[874992]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 14:02:01 rdma06 dnf[874992]: Metadata cache refreshed recently.
Apr 12 14:02:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 14:02:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 14:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 14:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 14:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 14:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 14:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 14:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:30:50 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 15:30:50 rdma06 dnf[878592]: Updating Subscription Management repositories.
Apr 12 15:30:50 rdma06 dnf[878592]: Unable to read consumer identity
Apr 12 15:30:50 rdma06 dnf[878592]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 15:30:52 rdma06 dnf[878592]: Copr repo for fmf owned by psss                 1.8 kB/s | 3.3 kB     00:01
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-AppStream                                7.0 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-AppStream-debuginfo                       52 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-BaseOS                                    52 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-BaseOS-debuginfo                          57 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-CRB                                       59 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-HighAvailability                          61 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-HighAvailability-debuginfo                58 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-NFV                                       55 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-NFV-debuginfo                             54 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-RT                                        62 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-RT':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-ResilientStorage                          57 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-ResilientStorage-debuginfo                57 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-SAP                                       61 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-SAP-debuginfo                             62 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-SAPHANA                                   60 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: beaker-SAPHANA-debuginfo                         55 kB/s | 168  B     00:00
Apr 12 15:30:52 rdma06 dnf[878592]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 12 15:30:52 rdma06 dnf[878592]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:52 rdma06 dnf[878592]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 15:30:53 rdma06 dnf[878592]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 12 15:30:54 rdma06 dnf[878592]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 12 15:30:54 rdma06 dnf[878592]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 12 15:30:54 rdma06 dnf[878592]: Metadata cache created.
Apr 12 15:30:54 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 15:30:54 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 15:30:54 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.040s CPU time.
Apr 12 15:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 15:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 15:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 15:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 15:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 15:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681330832.8690] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 12 16:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 16:53:01 rdma06 dnf[881938]: Updating Subscription Management repositories.
Apr 12 16:53:01 rdma06 dnf[881938]: Unable to read consumer identity
Apr 12 16:53:01 rdma06 dnf[881938]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 16:53:01 rdma06 dnf[881938]: Metadata cache refreshed recently.
Apr 12 16:53:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 16:53:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 16:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 16:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 16:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 16:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 16:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 16:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 17:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 17:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 17:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 17:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 17:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 18:26:01 rdma06 dnf[885702]: Updating Subscription Management repositories.
Apr 12 18:26:01 rdma06 dnf[885702]: Unable to read consumer identity
Apr 12 18:26:01 rdma06 dnf[885702]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 18:26:01 rdma06 dnf[885702]: Metadata cache refreshed recently.
Apr 12 18:26:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 18:26:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 18:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 18:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 18:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 18:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 18:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 18:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:59:16 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 19:59:17 rdma06 dnf[889475]: Updating Subscription Management repositories.
Apr 12 19:59:17 rdma06 dnf[889475]: Unable to read consumer identity
Apr 12 19:59:17 rdma06 dnf[889475]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 19:59:18 rdma06 dnf[889475]: Copr repo for fmf owned by psss                 3.9 kB/s | 3.3 kB     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-AppStream                                 46 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-AppStream-debuginfo                       52 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-BaseOS                                    54 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-BaseOS-debuginfo                          60 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-CRB                                       56 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-HighAvailability                          55 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-HighAvailability-debuginfo                55 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-NFV                                       57 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-NFV-debuginfo                             62 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-RT                                        55 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-RT':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-RT-debuginfo                              57 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-ResilientStorage                          59 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-ResilientStorage-debuginfo                61 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-SAP                                       64 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-SAPHANA                                   57 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: beaker-SAPHANA-debuginfo                         56 kB/s | 168  B     00:00
Apr 12 19:59:18 rdma06 dnf[889475]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 12 19:59:18 rdma06 dnf[889475]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:18 rdma06 dnf[889475]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 12 19:59:19 rdma06 dnf[889475]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 12 19:59:19 rdma06 dnf[889475]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 12 19:59:19 rdma06 dnf[889475]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 12 19:59:19 rdma06 dnf[889475]: Metadata cache created.
Apr 12 19:59:19 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 19:59:19 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 19:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 19:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 19:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 19:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 19:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 19:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 20:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 20:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 20:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 20:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 20:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:39:46 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 21:39:47 rdma06 dnf[893575]: Updating Subscription Management repositories.
Apr 12 21:39:47 rdma06 dnf[893575]: Unable to read consumer identity
Apr 12 21:39:47 rdma06 dnf[893575]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 21:39:47 rdma06 dnf[893575]: Metadata cache refreshed recently.
Apr 12 21:39:47 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 21:39:47 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 21:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 21:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 21:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 21:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 21:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 21:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:00:53 rdma06 sshd[894436]: main: sshd: ssh-rsa algorithm is disabled
Apr 12 22:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:00:59 rdma06 systemd-logind[1064]: New session 12 of user root.
Apr 12 22:00:59 rdma06 systemd[1]: Started Session 12 of User root.
Apr 12 22:01:00 rdma06 systemd[1]: Starting Hostname Service...
Apr 12 22:01:00 rdma06 systemd[1]: Started Hostname Service.
Apr 12 22:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:01:02 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr 12 22:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:01:30 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 12 22:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681352432.8434] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 12 22:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 12 22:41:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 12 22:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:20 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 12 22:44:20 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 12 22:44:20 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 12 22:44:20 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 12 22:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 12 22:54:01 rdma06 dnf[896653]: Updating Subscription Management repositories.
Apr 12 22:54:01 rdma06 dnf[896653]: Unable to read consumer identity
Apr 12 22:54:01 rdma06 dnf[896653]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 12 22:54:01 rdma06 dnf[896653]: Metadata cache refreshed recently.
Apr 12 22:54:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 12 22:54:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 12 22:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 22:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 22:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 22:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 22:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 22:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 12 23:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 12 23:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 12 23:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 12 23:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 12 23:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:00:00 rdma06 systemd[1]: Starting Rotate log files...
Apr 13 00:00:00 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 13 00:00:00 rdma06 systemd[1]: Finished Rotate log files.
Apr 13 00:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:51:50 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 00:51:51 rdma06 dnf[901478]: Updating Subscription Management repositories.
Apr 13 00:51:51 rdma06 dnf[901478]: Unable to read consumer identity
Apr 13 00:51:51 rdma06 dnf[901478]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 00:51:52 rdma06 dnf[901478]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-AppStream                                 49 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-AppStream-debuginfo                       57 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-BaseOS                                    61 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-BaseOS-debuginfo                          63 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-CRB                                       63 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-CRB-debuginfo                             64 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-HighAvailability                          58 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-HighAvailability-debuginfo                64 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-NFV                                       61 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-NFV-debuginfo                             56 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-RT                                        55 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-RT-debuginfo                              66 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-ResilientStorage                          60 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-ResilientStorage-debuginfo                66 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-SAP                                       55 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-SAP-debuginfo                             64 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-SAPHANA                                   62 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: beaker-SAPHANA-debuginfo                         60 kB/s | 168  B     00:00
Apr 13 00:51:52 rdma06 dnf[901478]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 00:51:52 rdma06 dnf[901478]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:52 rdma06 dnf[901478]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 00:51:53 rdma06 dnf[901478]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 00:51:54 rdma06 dnf[901478]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 00:51:54 rdma06 dnf[901478]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 00:51:54 rdma06 dnf[901478]: Metadata cache created.
Apr 13 00:51:54 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 00:51:54 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 00:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 00:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 00:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 00:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 00:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 00:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 01:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 01:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 01:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 01:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 01:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:35:14 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 02:35:15 rdma06 dnf[905688]: Updating Subscription Management repositories.
Apr 13 02:35:15 rdma06 dnf[905688]: Unable to read consumer identity
Apr 13 02:35:15 rdma06 dnf[905688]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 02:35:15 rdma06 dnf[905688]: Metadata cache refreshed recently.
Apr 13 02:35:15 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 02:35:15 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 02:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 02:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 02:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 02:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 02:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 02:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:43:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 03:43:01 rdma06 dnf[908444]: Updating Subscription Management repositories.
Apr 13 03:43:01 rdma06 dnf[908444]: Unable to read consumer identity
Apr 13 03:43:01 rdma06 dnf[908444]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 03:43:01 rdma06 dnf[908444]: Metadata cache refreshed recently.
Apr 13 03:43:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 03:43:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 03:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 03:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 03:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 03:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 03:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 03:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681374032.8632] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 13 04:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:54:50 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 04:54:51 rdma06 dnf[911448]: Updating Subscription Management repositories.
Apr 13 04:54:51 rdma06 dnf[911448]: Unable to read consumer identity
Apr 13 04:54:51 rdma06 dnf[911448]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 04:54:53 rdma06 dnf[911448]: Copr repo for fmf owned by psss                 2.0 kB/s | 3.3 kB     00:01
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-AppStream                                 54 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-BaseOS                                    65 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-BaseOS-debuginfo                          59 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-CRB                                       65 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-CRB-debuginfo                             61 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-HighAvailability                          62 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-HighAvailability-debuginfo                61 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-NFV                                       61 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-RT                                        53 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-ResilientStorage                          58 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-ResilientStorage-debuginfo                56 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-SAP                                       56 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-SAP-debuginfo                             52 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-SAPHANA                                   55 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-SAPHANA-debuginfo                         56 kB/s | 168  B     00:00
Apr 13 04:54:53 rdma06 dnf[911448]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 04:54:53 rdma06 dnf[911448]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 04:54:53 rdma06 dnf[911448]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 04:54:54 rdma06 dnf[911448]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 04:54:54 rdma06 dnf[911448]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 04:54:54 rdma06 dnf[911448]: Metadata cache created.
Apr 13 04:54:54 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 04:54:54 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 04:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 04:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 04:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 04:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 04:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 04:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 05:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 05:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 05:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 05:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 05:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:21:50 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 06:21:51 rdma06 dnf[915085]: Updating Subscription Management repositories.
Apr 13 06:21:51 rdma06 dnf[915085]: Unable to read consumer identity
Apr 13 06:21:51 rdma06 dnf[915085]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 06:21:51 rdma06 dnf[915085]: Metadata cache refreshed recently.
Apr 13 06:21:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 06:21:51 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 06:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 06:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 06:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 06:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 06:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 06:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 07:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 07:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 07:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 07:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 07:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:02:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 08:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:02:01 rdma06 dnf[919140]: Updating Subscription Management repositories.
Apr 13 08:02:01 rdma06 dnf[919140]: Unable to read consumer identity
Apr 13 08:02:01 rdma06 dnf[919140]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 08:02:02 rdma06 dnf[919140]: Copr repo for fmf owned by psss                 2.5 kB/s | 3.3 kB     00:01
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-AppStream                                 47 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-AppStream-debuginfo                       55 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-BaseOS                                    56 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-BaseOS-debuginfo                          54 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-CRB                                       63 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-CRB-debuginfo                             65 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-HighAvailability                          65 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-HighAvailability-debuginfo                65 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-NFV                                       61 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-NFV-debuginfo                             67 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-RT                                        54 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-RT-debuginfo                              56 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-ResilientStorage                          59 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-ResilientStorage-debuginfo                62 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-SAP                                       57 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-SAPHANA                                   56 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: beaker-SAPHANA-debuginfo                         56 kB/s | 168  B     00:00
Apr 13 08:02:02 rdma06 dnf[919140]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 08:02:02 rdma06 dnf[919140]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:02 rdma06 dnf[919140]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 08:02:03 rdma06 dnf[919140]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 08:02:04 rdma06 dnf[919140]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 08:02:04 rdma06 dnf[919140]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 08:02:04 rdma06 dnf[919140]: Metadata cache created.
Apr 13 08:02:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 08:02:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 08:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 08:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 08:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 08:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 08:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 08:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:03:50 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 09:03:51 rdma06 dnf[921666]: Updating Subscription Management repositories.
Apr 13 09:03:51 rdma06 dnf[921666]: Unable to read consumer identity
Apr 13 09:03:51 rdma06 dnf[921666]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 09:03:51 rdma06 dnf[921666]: Metadata cache refreshed recently.
Apr 13 09:03:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 09:03:51 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 09:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:49:48 rdma06 systemd[1]: session-12.scope: Deactivated successfully.
Apr 13 09:49:48 rdma06 systemd[1]: session-12.scope: Consumed 3min 5.110s CPU time.
Apr 13 09:49:48 rdma06 systemd-logind[1064]: Session 12 logged out. Waiting for processes to exit.
Apr 13 09:49:48 rdma06 systemd-logind[1064]: Removed session 12.
Apr 13 09:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:57:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 09:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 09:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 09:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 09:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 09:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:05:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:05:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:05:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:05:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:05:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:05:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:07:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:07:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:10:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:10:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:10:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:11:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:11:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:12:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:12:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:13:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:13:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:14:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:14:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:15:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:15:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:16:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:16:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:16:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:16:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:16:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:16:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:18:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:18:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:18:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:18:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:19:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:19:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:20:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681395632.9495] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 13 10:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:21:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:21:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:21:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:21:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:22:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:22:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:22:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:22:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:23:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:23:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:23:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:23:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:24:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:24:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:25:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:25:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:25:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:25:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:26:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:26:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:27:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 10:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:27:01 rdma06 dnf[925022]: Updating Subscription Management repositories.
Apr 13 10:27:01 rdma06 dnf[925022]: Unable to read consumer identity
Apr 13 10:27:01 rdma06 dnf[925022]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 10:27:01 rdma06 dnf[925022]: Metadata cache refreshed recently.
Apr 13 10:27:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 10:27:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 10:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:27:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:27:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:27:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:27:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:28:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:28:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:28:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:28:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:28:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:28:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:29:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:29:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:29:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:29:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:30:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:30:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:31:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:31:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:31:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:31:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:32:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:32:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:32:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:32:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:32:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:32:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:33:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:33:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:33:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:33:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:34:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:34:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:34:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:34:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:34:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:34:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:35:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:35:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:35:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:35:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:35:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:35:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:35:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:35:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:35:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:35:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:36:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:36:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:36:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:36:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:36:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:36:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:37:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:37:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:37:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:37:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:37:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:37:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:37:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:37:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:38:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:38:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:38:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:38:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:38:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:38:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:38:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:38:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:39:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:39:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:39:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:39:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:39:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:39:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:40:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:40:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:40:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:40:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:40:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:40:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:40:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:40:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:40:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:40:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:41:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:41:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:41:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:41:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:41:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:41:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:41:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:41:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:42:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:42:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:42:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:42:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:42:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:42:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:42:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:43:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:43:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:43:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:43:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:43:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:43:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:43:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:43:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:44:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:44:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:44:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:44:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:44:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:44:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:44:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:44:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:45:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:45:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:45:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:45:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:45:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:46:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:46:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:46:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:46:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:46:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:46:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:46:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:46:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:46:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:46:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:46:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:46:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:47:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:47:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:47:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:47:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:47:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:47:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:47:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:47:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:47:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:47:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:48:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:48:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:48:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:48:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:48:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:48:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:48:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:48:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:48:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:49:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:49:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:49:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:49:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:49:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:49:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:49:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:49:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:49:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:49:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:49:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:49:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:50:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:50:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:50:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:50:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:50:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:50:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:50:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:50:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:50:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:50:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:50:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:50:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:51:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:51:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:51:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:51:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:51:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:51:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:51:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:51:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:52:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:52:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:52:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:52:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:52:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:52:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:52:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:52:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:52:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:53:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:53:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:53:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:53:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:53:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:53:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:53:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:53:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:53:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:53:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:54:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:54:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:54:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:54:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:54:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:54:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:54:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:54:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:54:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:54:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:54:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:54:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:55:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:55:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:55:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:55:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:55:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:55:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:55:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:55:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:55:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:56:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:56:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:56:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:56:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:56:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:56:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:56:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:56:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:56:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:56:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:56:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:56:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:57:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:57:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:57:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:57:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:57:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:57:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:57:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:57:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:57:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:57:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:58:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:58:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:58:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:58:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:58:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:58:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:58:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:58:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:59:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:59:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:59:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:59:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:59:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:59:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:59:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:59:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 10:59:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:59:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:59:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 10:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 10:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 10:59:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 10:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:00:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:00:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:00:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:00:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:00:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:00:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:00:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:00:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:00:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:00:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:00:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:00:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:01:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:01:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:01:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:01:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:01:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:01:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:01:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:01:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:01:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:02:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:02:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:02:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:02:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:02:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:02:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:02:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:02:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:02:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:02:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:02:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:02:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:03:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:03:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:03:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:03:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:03:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:03:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:03:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:03:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:03:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:03:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:03:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:04:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:04:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:04:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:04:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:04:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:04:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:04:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:04:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:04:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:05:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:05:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:05:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:05:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:05:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:05:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:06:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:06:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:06:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:06:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:06:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:06:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:06:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:06:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:06:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:06:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:06:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:06:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:07:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:07:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:07:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:07:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:07:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:07:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:07:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:07:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:07:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:07:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:08:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:08:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:08:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:08:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:08:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:08:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:08:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:08:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:08:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:09:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:09:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:09:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:09:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:09:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:09:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:09:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:09:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:09:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:09:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:09:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:09:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:10:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:10:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:10:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:10:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:10:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:10:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:10:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:10:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:11:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:11:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:11:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:11:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:11:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:11:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:11:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:11:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:12:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:12:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:12:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:12:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:12:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:12:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:12:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:12:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:12:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:12:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:13:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:13:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:13:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:13:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:13:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:13:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:13:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:13:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:13:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:13:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:14:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:14:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:14:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:14:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:14:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:14:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:14:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:15:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:15:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:15:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:15:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:15:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:15:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:15:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:15:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:15:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:15:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:16:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:16:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:16:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:16:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:16:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:16:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:17:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:17:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:17:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:17:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:17:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:17:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:17:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:17:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:17:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:18:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:18:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:18:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:18:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:18:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:18:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:18:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:18:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:19:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:19:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:19:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:19:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:19:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:19:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:19:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:19:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:19:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:19:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:20:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:20:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:20:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:20:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:20:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:20:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:20:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:21:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:21:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:21:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:21:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:21:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:21:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:21:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:21:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:22:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:22:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:22:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:22:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:22:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:22:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:22:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:22:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:23:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:23:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:23:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:23:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:23:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:23:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:24:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:24:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:24:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:24:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:24:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:24:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:24:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:25:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:25:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:25:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:25:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:25:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:25:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:25:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:25:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:26:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:26:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:26:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:26:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:26:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:26:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:26:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:26:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:26:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:26:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:27:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:27:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:27:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:27:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:27:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:28:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:28:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:28:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:28:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:28:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:28:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:29:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:29:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:29:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:29:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:29:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:29:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:29:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:29:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:30:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:30:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:30:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:30:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:30:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:30:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:30:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:31:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:31:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:31:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:31:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:31:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:31:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:31:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:31:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:32:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:32:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:32:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:32:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:33:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:33:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:33:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:33:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:33:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:34:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:34:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:34:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:34:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:34:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:34:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:35:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:35:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:36:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:36:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:36:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:36:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:37:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:37:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:37:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:37:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:38:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:38:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:38:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:38:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:39:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:39:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:39:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:40:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:40:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:41:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:41:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:41:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:41:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:42:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:42:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:43:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:43:50 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:44:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:44:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:44:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:44:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:45:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:45:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:45:40 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:45:40 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:45:50 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 11:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:01 rdma06 dnf[928217]: Updating Subscription Management repositories.
Apr 13 11:46:01 rdma06 dnf[928217]: Unable to read consumer identity
Apr 13 11:46:01 rdma06 dnf[928217]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 11:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:46:03 rdma06 dnf[928217]: Copr repo for fmf owned by psss                 1.9 kB/s | 3.3 kB     00:01
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-AppStream                                9.1 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-BaseOS                                    63 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-BaseOS-debuginfo                          63 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-CRB                                       59 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-CRB-debuginfo                             62 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-HighAvailability                          62 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-HighAvailability-debuginfo                63 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-NFV                                       58 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-RT                                        55 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-RT-debuginfo                             5.7 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-ResilientStorage                          51 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-ResilientStorage-debuginfo                56 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-SAP                                       55 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-SAP-debuginfo                             63 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-SAPHANA                                   61 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-SAPHANA-debuginfo                         51 kB/s | 168  B     00:00
Apr 13 11:46:03 rdma06 dnf[928217]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 11:46:03 rdma06 dnf[928217]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 11:46:03 rdma06 dnf[928217]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 11:46:04 rdma06 dnf[928217]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 11:46:04 rdma06 dnf[928217]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 11:46:04 rdma06 dnf[928217]: Metadata cache created.
Apr 13 11:46:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 11:46:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 11:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:47:00 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:47:00 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:51:10 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:51:10 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:51:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:51:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:53:30 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:53:30 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:58:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:58:20 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 11:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 11:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 11:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 11:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 11:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:03:20 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:38:31 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 13 12:38:31 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 13 12:38:31 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 13 12:38:31 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 13 12:38:31 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 13 12:38:31 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 13 12:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 12:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 12:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 12:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 12:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 12:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:29:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 13:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:29:01 rdma06 dnf[932411]: Updating Subscription Management repositories.
Apr 13 13:29:01 rdma06 dnf[932411]: Unable to read consumer identity
Apr 13 13:29:01 rdma06 dnf[932411]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 13:29:01 rdma06 dnf[932411]: Metadata cache refreshed recently.
Apr 13 13:29:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 13:29:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 13:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 13:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 13:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 13:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 13:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 13:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 14:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:01 rdma06 dnf[935321]: Updating Subscription Management repositories.
Apr 13 14:41:01 rdma06 dnf[935321]: Unable to read consumer identity
Apr 13 14:41:01 rdma06 dnf[935321]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 14:41:01 rdma06 dnf[935321]: Metadata cache refreshed recently.
Apr 13 14:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 14:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 14:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 14:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 14:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 14:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 14:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 14:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 15:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 15:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 15:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 15:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 15:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 16:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:01 rdma06 dnf[939328]: Updating Subscription Management repositories.
Apr 13 16:20:01 rdma06 dnf[939328]: Unable to read consumer identity
Apr 13 16:20:01 rdma06 dnf[939328]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 16:20:02 rdma06 dnf[939328]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-AppStream                                 42 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-AppStream-debuginfo                       64 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-BaseOS                                    66 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-BaseOS-debuginfo                          65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-CRB                                       59 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-CRB-debuginfo                             61 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-HighAvailability                          58 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-HighAvailability-debuginfo                60 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-NFV                                       65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-NFV-debuginfo                             66 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-RT                                        63 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-RT-debuginfo                              65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-ResilientStorage                          65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-ResilientStorage-debuginfo                65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-SAP                                       65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-SAP-debuginfo                             24 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-SAPHANA                                   65 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: beaker-SAPHANA-debuginfo                         53 kB/s | 168  B     00:00
Apr 13 16:20:02 rdma06 dnf[939328]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 16:20:02 rdma06 dnf[939328]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:02 rdma06 dnf[939328]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 16:20:03 rdma06 dnf[939328]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 16:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:20:04 rdma06 dnf[939328]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 16:20:04 rdma06 dnf[939328]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 16:20:04 rdma06 dnf[939328]: Metadata cache created.
Apr 13 16:20:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 16:20:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 16:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681417232.8810] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 13 16:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 16:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 16:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 16:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 16:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 16:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:36:51 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 17:36:51 rdma06 dnf[942466]: Updating Subscription Management repositories.
Apr 13 17:36:51 rdma06 dnf[942466]: Unable to read consumer identity
Apr 13 17:36:51 rdma06 dnf[942466]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 17:36:51 rdma06 dnf[942466]: Metadata cache refreshed recently.
Apr 13 17:36:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 17:36:51 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 17:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:50:40 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr 13 17:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 17:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 17:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 17:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 17:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 17:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:57:51 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 18:57:51 rdma06 dnf[945749]: Updating Subscription Management repositories.
Apr 13 18:57:51 rdma06 dnf[945749]: Unable to read consumer identity
Apr 13 18:57:51 rdma06 dnf[945749]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 18:57:51 rdma06 dnf[945749]: Metadata cache refreshed recently.
Apr 13 18:57:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 18:57:51 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 18:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 18:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 18:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 18:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 18:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 18:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 19:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 19:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 19:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 19:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 19:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:16:59 rdma06 chronyd[1034]: Selected source 10.11.160.238
Apr 13 20:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:18:41 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 20:18:42 rdma06 dnf[949022]: Updating Subscription Management repositories.
Apr 13 20:18:42 rdma06 dnf[949022]: Unable to read consumer identity
Apr 13 20:18:42 rdma06 dnf[949022]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 20:18:43 rdma06 dnf[949022]: Copr repo for fmf owned by psss                 2.0 kB/s | 3.3 kB     00:01
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-AppStream                                 42 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-AppStream-debuginfo                       54 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-BaseOS                                    59 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-BaseOS-debuginfo                          62 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-CRB                                       64 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-CRB-debuginfo                             65 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-HighAvailability                          62 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-HighAvailability-debuginfo                67 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-NFV                                       66 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-NFV-debuginfo                             65 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-RT                                        63 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-RT-debuginfo                              66 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-ResilientStorage                          63 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-ResilientStorage-debuginfo                64 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-SAP                                       65 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-SAP-debuginfo                             65 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-SAPHANA                                   64 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: beaker-SAPHANA-debuginfo                         62 kB/s | 168  B     00:00
Apr 13 20:18:43 rdma06 dnf[949022]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 20:18:43 rdma06 dnf[949022]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:43 rdma06 dnf[949022]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 20:18:44 rdma06 dnf[949022]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 20:18:45 rdma06 dnf[949022]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 20:18:45 rdma06 dnf[949022]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 20:18:45 rdma06 dnf[949022]: Metadata cache created.
Apr 13 20:18:45 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 20:18:45 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 20:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 20:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 20:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 20:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 20:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 20:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 21:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 21:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 21:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 21:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 21:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:09:51 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 22:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:09:51 rdma06 dnf[953551]: Updating Subscription Management repositories.
Apr 13 22:09:51 rdma06 dnf[953551]: Unable to read consumer identity
Apr 13 22:09:51 rdma06 dnf[953551]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 22:09:51 rdma06 dnf[953551]: Metadata cache refreshed recently.
Apr 13 22:09:52 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 22:09:52 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 22:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681438832.8402] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 13 22:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:42:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 13 22:42:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 13 22:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:45:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 13 22:45:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 13 22:45:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 13 22:45:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 13 22:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 22:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 22:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 22:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 22:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 22:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:57:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 13 23:57:01 rdma06 dnf[957888]: Updating Subscription Management repositories.
Apr 13 23:57:01 rdma06 dnf[957888]: Unable to read consumer identity
Apr 13 23:57:01 rdma06 dnf[957888]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 13 23:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:02 rdma06 dnf[957888]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-AppStream                                 55 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-AppStream-debuginfo                       59 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-BaseOS                                    61 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-BaseOS-debuginfo                          65 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-CRB                                       65 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: beaker-HighAvailability                          64 kB/s | 168  B     00:00
Apr 13 23:57:02 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 13 23:57:02 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:02 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-HighAvailability-debuginfo                64 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-NFV                                       65 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-NFV-debuginfo                             59 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-RT                                        64 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-RT':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-RT-debuginfo                              64 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-ResilientStorage                          61 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-ResilientStorage-debuginfo                63 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-SAP                                       60 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-SAP-debuginfo                             61 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-SAPHANA                                   64 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-SAPHANA-debuginfo                         63 kB/s | 168  B     00:00
Apr 13 23:57:03 rdma06 dnf[957888]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 13 23:57:03 rdma06 dnf[957888]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 13 23:57:03 rdma06 dnf[957888]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 13 23:57:04 rdma06 dnf[957888]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 13 23:57:04 rdma06 dnf[957888]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 13 23:57:04 rdma06 dnf[957888]: Metadata cache created.
Apr 13 23:57:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 13 23:57:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 13 23:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 13 23:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 13 23:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 13 23:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 13 23:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 13 23:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:00:51 rdma06 systemd[1]: Starting Rotate log files...
Apr 14 00:00:51 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 14 00:00:51 rdma06 systemd[1]: Finished Rotate log files.
Apr 14 00:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 00:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 00:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 00:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 00:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 00:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:06:51 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 01:06:52 rdma06 dnf[960762]: Updating Subscription Management repositories.
Apr 14 01:06:52 rdma06 dnf[960762]: Unable to read consumer identity
Apr 14 01:06:52 rdma06 dnf[960762]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 01:06:52 rdma06 dnf[960762]: Metadata cache refreshed recently.
Apr 14 01:06:52 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 01:06:52 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 01:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 01:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 01:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 01:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 01:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 01:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 02:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 02:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 02:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 02:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 02:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 03:05:01 rdma06 dnf[965562]: Updating Subscription Management repositories.
Apr 14 03:05:01 rdma06 dnf[965562]: Unable to read consumer identity
Apr 14 03:05:01 rdma06 dnf[965562]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 03:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:02 rdma06 dnf[965562]: Copr repo for fmf owned by psss                 6.1 kB/s | 3.3 kB     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-AppStream                                 52 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-AppStream-debuginfo                       57 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-BaseOS                                    54 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-BaseOS-debuginfo                          56 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-HighAvailability                          63 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-HighAvailability-debuginfo                58 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-NFV                                       62 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-NFV-debuginfo                             50 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-RT                                        63 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-RT':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-RT-debuginfo                              63 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-ResilientStorage                          63 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-ResilientStorage-debuginfo                65 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-SAP                                       54 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-SAP-debuginfo                             54 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-SAPHANA                                   56 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-SAPHANA-debuginfo                         57 kB/s | 168  B     00:00
Apr 14 03:05:02 rdma06 dnf[965562]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 14 03:05:02 rdma06 dnf[965562]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 03:05:02 rdma06 dnf[965562]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 14 03:05:03 rdma06 dnf[965562]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 14 03:05:03 rdma06 dnf[965562]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 14 03:05:03 rdma06 dnf[965562]: Metadata cache created.
Apr 14 03:05:03 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 03:05:03 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 03:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:17 rdma06 sshd[965954]: main: sshd: ssh-rsa algorithm is disabled
Apr 14 03:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:14:29 rdma06 systemd-logind[1064]: New session 13 of user root.
Apr 14 03:14:29 rdma06 systemd[1]: Started Session 13 of User root.
Apr 14 03:14:30 rdma06 systemd[1]: Starting Hostname Service...
Apr 14 03:14:30 rdma06 systemd[1]: Started Hostname Service.
Apr 14 03:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:35 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr 14 03:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:15:00 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 14 03:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 03:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 03:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 03:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 03:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 03:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681460432.8106] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 14 04:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 04:50:01 rdma06 dnf[969889]: Updating Subscription Management repositories.
Apr 14 04:50:01 rdma06 dnf[969889]: Unable to read consumer identity
Apr 14 04:50:01 rdma06 dnf[969889]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 04:50:01 rdma06 dnf[969889]: Metadata cache refreshed recently.
Apr 14 04:50:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 04:50:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 04:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 04:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 04:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 04:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 04:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 04:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 05:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 05:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 05:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 05:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 05:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 06:03:01 rdma06 dnf[972849]: Updating Subscription Management repositories.
Apr 14 06:03:01 rdma06 dnf[972849]: Unable to read consumer identity
Apr 14 06:03:01 rdma06 dnf[972849]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 06:03:01 rdma06 dnf[972849]: Metadata cache refreshed recently.
Apr 14 06:03:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 06:03:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 06:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 06:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 06:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 06:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 06:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 06:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:32:21 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 07:32:22 rdma06 dnf[976468]: Updating Subscription Management repositories.
Apr 14 07:32:22 rdma06 dnf[976468]: Unable to read consumer identity
Apr 14 07:32:22 rdma06 dnf[976468]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 07:32:24 rdma06 dnf[976468]: Copr repo for fmf owned by psss                 1.8 kB/s | 3.3 kB     00:01
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-AppStream                                 49 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-AppStream-debuginfo                       54 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-BaseOS                                    55 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-BaseOS-debuginfo                          58 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-CRB                                       65 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-CRB-debuginfo                             64 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-HighAvailability                          65 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-HighAvailability-debuginfo                64 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-NFV                                       73 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-NFV-debuginfo                             71 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-RT                                        61 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-RT':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-ResilientStorage                          54 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-ResilientStorage-debuginfo                58 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-SAP                                       60 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-SAPHANA                                   66 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-SAPHANA-debuginfo                         60 kB/s | 168  B     00:00
Apr 14 07:32:24 rdma06 dnf[976468]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 14 07:32:24 rdma06 dnf[976468]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 07:32:24 rdma06 dnf[976468]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 14 07:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:32:25 rdma06 dnf[976468]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 14 07:32:25 rdma06 dnf[976468]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 14 07:32:25 rdma06 dnf[976468]: Metadata cache created.
Apr 14 07:32:25 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 07:32:25 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 07:32:25 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.021s CPU time.
Apr 14 07:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 07:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 07:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 07:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 07:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 07:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 08:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 08:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 08:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 08:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 08:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:12:51 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 09:12:52 rdma06 dnf[980565]: Updating Subscription Management repositories.
Apr 14 09:12:52 rdma06 dnf[980565]: Unable to read consumer identity
Apr 14 09:12:52 rdma06 dnf[980565]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 09:12:52 rdma06 dnf[980565]: Metadata cache refreshed recently.
Apr 14 09:12:52 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 09:12:52 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 09:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 09:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 09:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 09:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 09:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 09:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681482032.8331] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 14 10:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:38:58 rdma06 chronyd[1034]: Selected source 10.13.199.1
Apr 14 10:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:57:51 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 10:57:52 rdma06 dnf[984814]: Updating Subscription Management repositories.
Apr 14 10:57:52 rdma06 dnf[984814]: Unable to read consumer identity
Apr 14 10:57:52 rdma06 dnf[984814]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 10:57:53 rdma06 dnf[984814]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-AppStream                                 53 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-AppStream-debuginfo                       12 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-BaseOS                                    57 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-BaseOS-debuginfo                          58 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-CRB-debuginfo                             57 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-HighAvailability                          57 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-HighAvailability-debuginfo                59 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-NFV                                       61 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-NFV-debuginfo                             59 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: beaker-RT                                        52 kB/s | 168  B     00:00
Apr 14 10:57:53 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-RT':
Apr 14 10:57:53 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:53 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-RT-debuginfo                              54 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-ResilientStorage                          56 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-ResilientStorage-debuginfo                57 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-SAP                                       57 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-SAP-debuginfo                             63 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-SAPHANA                                   58 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-SAPHANA-debuginfo                         64 kB/s | 168  B     00:00
Apr 14 10:57:54 rdma06 dnf[984814]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 14 10:57:54 rdma06 dnf[984814]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 10:57:54 rdma06 dnf[984814]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 14 10:57:55 rdma06 dnf[984814]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 14 10:57:55 rdma06 dnf[984814]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 14 10:57:55 rdma06 dnf[984814]: Metadata cache created.
Apr 14 10:57:55 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 10:57:55 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 10:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 10:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 10:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 10:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 10:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 10:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 11:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 11:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 11:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 11:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 11:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 12:23:01 rdma06 dnf[988288]: Updating Subscription Management repositories.
Apr 14 12:23:01 rdma06 dnf[988288]: Unable to read consumer identity
Apr 14 12:23:01 rdma06 dnf[988288]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 12:23:01 rdma06 dnf[988288]: Metadata cache refreshed recently.
Apr 14 12:23:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 12:23:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 12:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 12:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 12:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 12:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 12:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 12:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:11:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:13:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:13:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:13:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:13:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:15:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:15:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:16:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:16:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:19:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:19:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:21:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:21:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:21:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:23:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:23:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 13:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:24:01 rdma06 dnf[990766]: Updating Subscription Management repositories.
Apr 14 13:24:01 rdma06 dnf[990766]: Unable to read consumer identity
Apr 14 13:24:01 rdma06 dnf[990766]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 13:24:01 rdma06 dnf[990766]: Metadata cache refreshed recently.
Apr 14 13:24:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 13:24:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 13:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:26:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:26:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:27:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:28:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:28:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:28:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:28:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:31:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:31:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:31:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:31:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:32:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:32:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:33:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:33:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:33:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:33:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:34:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:34:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:35:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:35:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:35:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:35:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:35:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:35:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:38:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:38:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:38:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:38:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:38:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:38:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:40:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:40:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:41:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:41:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:42:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:42:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:43:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:43:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:43:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:43:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:43:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:43:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:43:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:44:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:44:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:45:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:45:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:45:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:45:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:45:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:45:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:46:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:46:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:46:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:46:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:46:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:46:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:47:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:47:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:47:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:47:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:47:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:47:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:48:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:48:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:48:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:48:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:48:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:48:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:48:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:49:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:49:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:49:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:49:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:49:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:49:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:50:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:50:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:50:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:50:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:50:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:50:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:51:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:51:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:51:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:51:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:51:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:51:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:51:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:52:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:52:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:52:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:52:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:52:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:52:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:52:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:53:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:53:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:53:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:53:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:53:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:53:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:53:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:53:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:53:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:53:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:54:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:54:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:54:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:54:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:54:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:54:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:54:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:54:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:54:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:54:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:54:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:54:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:55:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:55:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:55:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:55:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:56:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:56:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:56:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:56:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:56:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:56:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:56:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:56:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:56:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:57:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:57:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:57:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:57:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:57:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:57:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:57:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:57:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:57:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:58:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:58:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:58:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:58:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:58:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:58:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:58:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:58:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:58:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:58:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:58:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:58:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:59:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:59:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:59:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:59:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:59:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 13:59:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:59:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 13:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 13:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 13:59:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 13:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:00:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:00:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:00:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:00:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:00:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:00:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:00:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:00:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:00:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:00:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:01:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:01:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:01:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:01:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:01:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:01:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:01:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:01:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:01:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:01:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:01:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:01:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:02:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:02:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:02:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:02:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:02:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:02:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:02:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:02:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:02:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:03:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:03:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:03:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:03:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:03:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:03:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:03:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:03:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:03:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:03:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:03:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:03:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:04:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:04:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:04:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:04:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:04:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:04:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:04:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:04:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:04:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:04:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:05:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:05:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:05:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:05:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:05:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:05:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:05:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:05:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:05:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:06:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:06:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:06:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:06:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:06:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:06:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:06:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:06:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:06:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:06:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:06:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:06:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:07:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:07:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:07:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:07:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:07:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:07:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:07:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:07:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:07:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:07:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:08:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:08:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:08:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:08:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:08:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:08:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:08:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:08:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:08:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:09:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:09:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:09:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:09:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:09:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:09:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:09:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:09:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:09:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:09:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:09:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:09:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:10:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:10:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:10:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:10:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:10:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:10:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:10:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:10:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:10:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:10:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:10:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:10:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:11:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:11:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:11:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:11:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:11:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:11:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:11:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:11:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:11:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:11:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:12:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:12:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:12:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:12:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:12:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:12:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:12:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:12:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:12:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:13:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:13:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:13:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:13:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:13:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:13:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:13:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:13:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:14:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:14:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:14:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:14:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:14:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:14:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:14:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:14:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:14:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:14:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:14:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:14:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:15:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:15:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:15:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:15:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:15:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:15:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:15:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:15:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:16:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:16:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:16:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:16:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:16:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:16:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:16:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:16:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:16:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:16:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:17:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:17:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:17:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:17:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:17:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:17:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:17:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:17:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:17:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:17:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:17:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:17:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:18:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:18:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:18:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:18:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:18:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:18:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:18:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:18:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:18:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:19:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:19:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:19:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:19:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:19:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:19:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:19:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:19:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:19:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:19:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:20:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:20:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:20:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:20:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:20:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:20:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:20:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:20:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:20:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:20:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:20:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:20:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:21:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:21:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:21:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:21:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:21:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:22:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:22:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:22:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:22:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:22:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:22:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:22:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:22:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:22:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:22:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:22:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:22:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:23:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:23:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:23:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:23:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:23:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:23:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:23:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:23:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:23:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:23:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:24:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:24:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:24:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:24:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:24:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:24:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:24:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:24:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:24:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:25:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:25:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:25:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:25:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:25:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:25:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:25:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:25:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:25:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:25:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:25:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:25:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:26:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:26:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:26:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:26:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:26:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:26:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:26:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:26:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:26:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:26:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:27:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:27:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:27:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:27:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:27:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:27:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:27:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:27:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:27:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:27:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:28:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:28:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:28:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:28:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:28:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:29:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:29:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:29:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:29:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:29:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:29:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:29:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:29:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:29:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:29:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:29:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:29:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:30:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:30:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:30:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:30:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:30:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:30:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:30:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:30:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:30:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:30:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:30:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:30:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:31:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:31:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:31:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:31:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:31:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:31:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:32:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:32:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:32:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:32:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:32:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:32:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:32:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:32:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:32:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:32:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:33:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:33:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:33:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:33:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:33:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:33:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:34:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:34:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:34:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:34:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:34:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:34:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:34:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:35:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:35:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:35:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:35:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:35:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:35:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:36:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:36:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:36:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:36:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:36:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:36:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:36:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:36:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:36:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:36:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:36:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:36:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:37:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:37:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:37:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:37:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:37:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:37:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:37:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:37:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:37:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:38:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:38:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:38:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:38:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:38:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:38:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:39:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 14:39:01 rdma06 dnf[993803]: Updating Subscription Management repositories.
Apr 14 14:39:01 rdma06 dnf[993803]: Unable to read consumer identity
Apr 14 14:39:01 rdma06 dnf[993803]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 14:39:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:39:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:39:02 rdma06 dnf[993803]: Copr repo for fmf owned by psss                 3.7 kB/s | 3.3 kB     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-AppStream                                 49 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-BaseOS                                    56 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-BaseOS-debuginfo                          56 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-CRB                                       64 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-CRB-debuginfo                             55 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-HighAvailability                          67 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-HighAvailability-debuginfo                59 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-NFV                                       63 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-NFV-debuginfo                             55 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-RT                                        60 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-RT':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-ResilientStorage                          65 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-ResilientStorage-debuginfo                58 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-SAP                                       57 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-SAP-debuginfo                             58 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-SAPHANA                                   65 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: beaker-SAPHANA-debuginfo                         61 kB/s | 168  B     00:00
Apr 14 14:39:02 rdma06 dnf[993803]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 14 14:39:02 rdma06 dnf[993803]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:02 rdma06 dnf[993803]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 14:39:03 rdma06 dnf[993803]: beaker-harness                                  1.7 kB/s | 1.3 kB     00:00
Apr 14 14:39:03 rdma06 dnf[993803]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 14 14:39:03 rdma06 dnf[993803]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 14 14:39:03 rdma06 dnf[993803]: Metadata cache created.
Apr 14 14:39:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 14:39:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 14:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:39:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:39:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:39:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:39:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:39:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:39:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:39:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:39:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:39:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:39:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:40:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:40:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:40:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:40:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:40:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:40:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:40:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:40:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:41:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:41:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:41:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:41:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:41:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:41:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:41:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:41:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:41:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:41:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:42:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:42:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:42:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:42:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:42:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:42:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:42:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:42:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:42:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:42:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:43:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:43:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:44:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:44:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:44:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:44:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:44:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:44:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:44:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:44:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:44:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:44:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:45:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:45:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:45:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:45:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:45:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:45:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:46:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:46:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:46:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:46:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:47:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:47:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:47:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:47:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:47:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:48:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:48:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:48:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:48:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:48:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:49:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:49:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:49:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:49:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:50:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:50:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:50:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:50:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:50:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:51:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:51:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:51:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:51:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:51:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:52:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:52:01 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:52:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:52:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:53:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:53:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:55:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:55:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:55:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:55:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:55:41 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:55:41 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:55:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:55:51 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:51 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:57:01 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:57:31 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:57:31 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:59:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:59:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 14:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 14:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 14:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 14:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 14:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:00:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:00:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:04:11 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:04:11 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:07:21 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:07:21 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 15:41:01 rdma06 dnf[996332]: Updating Subscription Management repositories.
Apr 14 15:41:01 rdma06 dnf[996332]: Unable to read consumer identity
Apr 14 15:41:01 rdma06 dnf[996332]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 15:41:01 rdma06 dnf[996332]: Metadata cache refreshed recently.
Apr 14 15:41:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 15:41:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 15:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 15:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 15:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 15:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 15:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 15:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681503632.8364] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 14 16:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 16:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 16:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 16:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 16:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 16:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:20:19 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 17:20:20 rdma06 dnf[1000358]: Updating Subscription Management repositories.
Apr 14 17:20:20 rdma06 dnf[1000358]: Unable to read consumer identity
Apr 14 17:20:20 rdma06 dnf[1000358]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 17:20:20 rdma06 dnf[1000358]: Metadata cache refreshed recently.
Apr 14 17:20:20 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 17:20:20 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 17:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 17:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 17:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 17:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 17:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 17:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 18:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 18:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 18:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 18:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 18:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 19:18:01 rdma06 dnf[1005129]: Updating Subscription Management repositories.
Apr 14 19:18:01 rdma06 dnf[1005129]: Unable to read consumer identity
Apr 14 19:18:01 rdma06 dnf[1005129]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 19:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:02 rdma06 dnf[1005129]: Copr repo for fmf owned by psss                 2.8 kB/s | 3.3 kB     00:01
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-AppStream                                 44 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-AppStream-debuginfo                       49 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-BaseOS                                    61 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-BaseOS-debuginfo                          58 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-CRB                                       58 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-CRB-debuginfo                             61 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-HighAvailability                          68 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-HighAvailability-debuginfo                60 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-NFV                                       62 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-RT                                        56 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-RT':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-RT-debuginfo                              53 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-ResilientStorage                          55 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-ResilientStorage-debuginfo                59 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-SAP                                       57 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-SAP-debuginfo                             55 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-SAPHANA                                   55 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: beaker-SAPHANA-debuginfo                         62 kB/s | 168  B     00:00
Apr 14 19:18:02 rdma06 dnf[1005129]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 14 19:18:02 rdma06 dnf[1005129]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:02 rdma06 dnf[1005129]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 19:18:03 rdma06 dnf[1005129]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 14 19:18:03 rdma06 dnf[1005129]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 14 19:18:04 rdma06 dnf[1005129]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 14 19:18:04 rdma06 dnf[1005129]: Metadata cache created.
Apr 14 19:18:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 19:18:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 19:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 19:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 19:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 19:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 19:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 19:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 20:20:01 rdma06 dnf[1007661]: Updating Subscription Management repositories.
Apr 14 20:20:01 rdma06 dnf[1007661]: Unable to read consumer identity
Apr 14 20:20:01 rdma06 dnf[1007661]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 20:20:01 rdma06 dnf[1007661]: Metadata cache refreshed recently.
Apr 14 20:20:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 20:20:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 20:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 20:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 20:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 20:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 20:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 20:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 21:53:01 rdma06 dnf[1011423]: Updating Subscription Management repositories.
Apr 14 21:53:01 rdma06 dnf[1011423]: Unable to read consumer identity
Apr 14 21:53:01 rdma06 dnf[1011423]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 21:53:01 rdma06 dnf[1011423]: Metadata cache refreshed recently.
Apr 14 21:53:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 21:53:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 21:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 21:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 21:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 21:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 21:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 21:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681525232.8404] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 14 22:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:43:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 14 22:43:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 14 22:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:45:52 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 14 22:45:52 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 14 22:45:52 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 14 22:45:52 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 14 22:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 22:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 22:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 22:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 22:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 22:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:08:56 rdma06 systemd-logind[1064]: Session 13 logged out. Waiting for processes to exit.
Apr 14 23:08:56 rdma06 systemd[1]: session-13.scope: Deactivated successfully.
Apr 14 23:08:56 rdma06 systemd-logind[1064]: Removed session 13.
Apr 14 23:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:13:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 14 23:13:01 rdma06 dnf[1014686]: Updating Subscription Management repositories.
Apr 14 23:13:01 rdma06 dnf[1014686]: Unable to read consumer identity
Apr 14 23:13:01 rdma06 dnf[1014686]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 14 23:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:13:02 rdma06 dnf[1014686]: Copr repo for fmf owned by psss                 3.6 kB/s | 3.3 kB     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-AppStream                                 38 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-BaseOS                                    52 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-BaseOS-debuginfo                          56 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-CRB                                       56 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-CRB-debuginfo                             56 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-HighAvailability                          61 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-HighAvailability-debuginfo                57 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-NFV                                       55 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-NFV-debuginfo                             60 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-RT                                        64 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-RT':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-RT-debuginfo                              61 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-ResilientStorage                          56 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-ResilientStorage-debuginfo                61 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-SAP                                       60 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-SAP-debuginfo                            2.6 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-SAPHANA                                   64 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: beaker-SAPHANA-debuginfo                         65 kB/s | 168  B     00:00
Apr 14 23:13:02 rdma06 dnf[1014686]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 14 23:13:02 rdma06 dnf[1014686]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:02 rdma06 dnf[1014686]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 14 23:13:03 rdma06 dnf[1014686]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 14 23:13:03 rdma06 dnf[1014686]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 14 23:13:03 rdma06 dnf[1014686]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 14 23:13:03 rdma06 dnf[1014686]: Metadata cache created.
Apr 14 23:13:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 14 23:13:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 14 23:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 14 23:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 14 23:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 14 23:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 14 23:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 14 23:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:00:49 rdma06 systemd[1]: Starting Rotate log files...
Apr 15 00:00:49 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 15 00:00:49 rdma06 systemd[1]: Finished Rotate log files.
Apr 15 00:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 00:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 00:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 00:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 00:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 00:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:06:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 01:06:01 rdma06 dnf[1019303]: Updating Subscription Management repositories.
Apr 15 01:06:01 rdma06 dnf[1019303]: Unable to read consumer identity
Apr 15 01:06:01 rdma06 dnf[1019303]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 01:06:01 rdma06 dnf[1019303]: Metadata cache refreshed recently.
Apr 15 01:06:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 01:06:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 01:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 01:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 01:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 01:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 01:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 01:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:22 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 02:41:22 rdma06 dnf[1023165]: Updating Subscription Management repositories.
Apr 15 02:41:22 rdma06 dnf[1023165]: Unable to read consumer identity
Apr 15 02:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:41:22 rdma06 dnf[1023165]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 02:41:24 rdma06 dnf[1023165]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-AppStream                                 46 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-AppStream-debuginfo                       11 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-BaseOS                                    59 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-BaseOS-debuginfo                          57 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-CRB                                       54 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-CRB-debuginfo                             63 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-HighAvailability                          57 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-HighAvailability-debuginfo                66 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-NFV                                       52 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-NFV-debuginfo                             55 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-RT                                        56 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-ResilientStorage                          59 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-ResilientStorage-debuginfo                49 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-SAP                                       60 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-SAP-debuginfo                             55 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-SAPHANA                                   55 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: beaker-SAPHANA-debuginfo                         57 kB/s | 168  B     00:00
Apr 15 02:41:24 rdma06 dnf[1023165]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 02:41:24 rdma06 dnf[1023165]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:24 rdma06 dnf[1023165]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 02:41:25 rdma06 dnf[1023165]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 02:41:25 rdma06 dnf[1023165]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 02:41:26 rdma06 dnf[1023165]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 02:41:26 rdma06 dnf[1023165]: Metadata cache created.
Apr 15 02:41:26 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 02:41:26 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 02:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 02:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 02:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 02:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 02:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 02:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 03:42:01 rdma06 dnf[1025645]: Updating Subscription Management repositories.
Apr 15 03:42:01 rdma06 dnf[1025645]: Unable to read consumer identity
Apr 15 03:42:01 rdma06 dnf[1025645]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 03:42:01 rdma06 dnf[1025645]: Metadata cache refreshed recently.
Apr 15 03:42:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 03:42:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 03:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 03:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 03:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 03:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 03:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 03:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681546832.8492] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 15 04:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 04:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 04:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 04:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 04:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 04:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:00:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 05:00:49 rdma06 dnf[1028825]: Updating Subscription Management repositories.
Apr 15 05:00:49 rdma06 dnf[1028825]: Unable to read consumer identity
Apr 15 05:00:49 rdma06 dnf[1028825]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 05:00:49 rdma06 dnf[1028825]: Metadata cache refreshed recently.
Apr 15 05:00:49 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 05:00:49 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 05:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 05:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 05:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 05:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 05:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 05:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:04:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 06:04:01 rdma06 dnf[1031399]: Updating Subscription Management repositories.
Apr 15 06:04:01 rdma06 dnf[1031399]: Unable to read consumer identity
Apr 15 06:04:01 rdma06 dnf[1031399]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 06:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:04:03 rdma06 dnf[1031399]: Copr repo for fmf owned by psss                 2.0 kB/s | 3.3 kB     00:01
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-AppStream                                 49 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-AppStream-debuginfo                       53 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-BaseOS                                    39 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-BaseOS-debuginfo                          21 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-CRB                                       56 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-CRB-debuginfo                             56 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-HighAvailability                          59 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-HighAvailability-debuginfo                51 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-NFV                                       61 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-NFV-debuginfo                             57 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-RT                                        57 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-RT-debuginfo                              56 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-ResilientStorage                          55 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-ResilientStorage-debuginfo                17 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-SAP                                       36 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-SAP-debuginfo                             58 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-SAPHANA                                   51 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-SAPHANA-debuginfo                         58 kB/s | 168  B     00:00
Apr 15 06:04:03 rdma06 dnf[1031399]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 06:04:03 rdma06 dnf[1031399]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 06:04:03 rdma06 dnf[1031399]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 06:04:04 rdma06 dnf[1031399]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 06:04:04 rdma06 dnf[1031399]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 06:04:04 rdma06 dnf[1031399]: Metadata cache created.
Apr 15 06:04:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 06:04:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 06:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 06:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 06:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 06:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 06:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 06:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:33:49 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 07:33:50 rdma06 dnf[1035048]: Updating Subscription Management repositories.
Apr 15 07:33:50 rdma06 dnf[1035048]: Unable to read consumer identity
Apr 15 07:33:50 rdma06 dnf[1035048]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 07:33:50 rdma06 dnf[1035048]: Metadata cache refreshed recently.
Apr 15 07:33:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 07:33:50 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 07:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 07:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 07:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 07:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 07:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 07:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:54:52 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 08:54:53 rdma06 dnf[1038325]: Updating Subscription Management repositories.
Apr 15 08:54:53 rdma06 dnf[1038325]: Unable to read consumer identity
Apr 15 08:54:53 rdma06 dnf[1038325]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 08:54:53 rdma06 dnf[1038325]: Metadata cache refreshed recently.
Apr 15 08:54:53 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 08:54:53 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 08:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 08:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 08:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 08:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 08:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 08:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:21:12 rdma06 chronyd[1034]: Selected source 10.18.100.10
Apr 15 09:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 09:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 09:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 09:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 09:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 09:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681568432.8315] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 15 10:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:47:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 10:47:01 rdma06 dnf[1042857]: Updating Subscription Management repositories.
Apr 15 10:47:01 rdma06 dnf[1042857]: Unable to read consumer identity
Apr 15 10:47:01 rdma06 dnf[1042857]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 10:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:47:03 rdma06 dnf[1042857]: Copr repo for fmf owned by psss                 1.7 kB/s | 3.3 kB     00:01
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-AppStream                                 50 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-AppStream-debuginfo                       53 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-BaseOS                                    59 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-BaseOS-debuginfo                          55 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-CRB                                       59 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-CRB-debuginfo                             66 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-HighAvailability                          65 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-HighAvailability-debuginfo                55 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-NFV                                       60 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-NFV-debuginfo                             63 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-RT                                        64 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-RT-debuginfo                              58 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-ResilientStorage                          55 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-ResilientStorage-debuginfo                59 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-SAP                                       60 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-SAP-debuginfo                             57 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-SAPHANA                                   58 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: beaker-SAPHANA-debuginfo                         52 kB/s | 168  B     00:00
Apr 15 10:47:03 rdma06 dnf[1042857]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 10:47:03 rdma06 dnf[1042857]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:03 rdma06 dnf[1042857]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 10:47:04 rdma06 dnf[1042857]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 10:47:04 rdma06 dnf[1042857]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 10:47:04 rdma06 dnf[1042857]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 10:47:04 rdma06 dnf[1042857]: Metadata cache created.
Apr 15 10:47:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 10:47:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 10:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 10:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 10:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 10:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 10:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 10:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 11:54:01 rdma06 dnf[1045591]: Updating Subscription Management repositories.
Apr 15 11:54:01 rdma06 dnf[1045591]: Unable to read consumer identity
Apr 15 11:54:01 rdma06 dnf[1045591]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 11:54:01 rdma06 dnf[1045591]: Metadata cache refreshed recently.
Apr 15 11:54:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 11:54:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 11:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 11:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 11:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 11:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 11:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 11:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 12:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 12:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 12:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 12:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 12:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 13:48:01 rdma06 dnf[1050209]: Updating Subscription Management repositories.
Apr 15 13:48:01 rdma06 dnf[1050209]: Unable to read consumer identity
Apr 15 13:48:01 rdma06 dnf[1050209]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 13:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:02 rdma06 dnf[1050209]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr 15 13:48:02 rdma06 dnf[1050209]: beaker-AppStream                                 13 kB/s | 168  B     00:00
Apr 15 13:48:02 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 13:48:02 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:02 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:02 rdma06 dnf[1050209]: beaker-AppStream-debuginfo                      5.6 kB/s | 168  B     00:00
Apr 15 13:48:02 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 13:48:02 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:02 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:02 rdma06 dnf[1050209]: beaker-BaseOS                                   5.3 kB/s | 168  B     00:00
Apr 15 13:48:02 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 13:48:02 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:02 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-BaseOS-debuginfo                         3.7 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-CRB                                      9.1 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-CRB-debuginfo                            3.0 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-HighAvailability                          13 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-HighAvailability-debuginfo                65 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-NFV                                       55 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-NFV-debuginfo                             54 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-RT                                        55 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-ResilientStorage                          58 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-ResilientStorage-debuginfo                55 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-SAP                                       59 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-SAP-debuginfo                             52 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-SAPHANA                                   56 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-SAPHANA-debuginfo                         62 kB/s | 168  B     00:00
Apr 15 13:48:03 rdma06 dnf[1050209]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 13:48:03 rdma06 dnf[1050209]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 13:48:03 rdma06 dnf[1050209]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 13:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:48:04 rdma06 dnf[1050209]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 13:48:04 rdma06 dnf[1050209]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 13:48:04 rdma06 dnf[1050209]: Metadata cache created.
Apr 15 13:48:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 13:48:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 13:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 13:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 13:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 13:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 13:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 13:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 14:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 14:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 14:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 14:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 14:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:30:52 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 15:30:53 rdma06 dnf[1054398]: Updating Subscription Management repositories.
Apr 15 15:30:53 rdma06 dnf[1054398]: Unable to read consumer identity
Apr 15 15:30:53 rdma06 dnf[1054398]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 15:30:53 rdma06 dnf[1054398]: Metadata cache refreshed recently.
Apr 15 15:30:53 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 15:30:53 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 15:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 15:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 15:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 15:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 15:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 15:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:03:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:03:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:06:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:06:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:09:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:09:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:13:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:13:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:14:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:14:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:16:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:16:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:16:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:16:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681590032.7892] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 15 16:20:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:20:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:20:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:20:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:23:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:23:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:23:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:23:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:26:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:26:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:29:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:29:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:30:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:30:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:30:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:30:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:32:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:32:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:32:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:32:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:33:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:33:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:34:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:34:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:34:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:34:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:35:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:35:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:35:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:37:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:38:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:38:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:38:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:38:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:38:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:38:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:39:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:39:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:39:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:39:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:40:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:40:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:40:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:41:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:41:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:42:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:42:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:42:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:42:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:42:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:42:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:43:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:43:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:44:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:44:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:44:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:44:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:44:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:45:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:45:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:45:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:45:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:46:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:46:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:46:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:46:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:47:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:47:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:47:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:47:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:47:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:47:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:48:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:48:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:48:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:48:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:48:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:48:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:48:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:48:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:49:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:49:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:49:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:49:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:49:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:49:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:50:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:50:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:50:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:50:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:50:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:50:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:51:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:51:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:51:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:51:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:51:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:51:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:52:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:52:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:52:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:52:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:52:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:52:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:53:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:53:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:53:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:53:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:54:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:54:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:54:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:54:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:54:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:54:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:55:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:55:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:55:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:55:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:55:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:55:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:55:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:55:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:55:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:55:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:56:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:56:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:56:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:56:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:56:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:57:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:57:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:57:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:57:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:57:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:57:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:57:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:57:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:58:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:58:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:58:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:58:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:58:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:58:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:58:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:59:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:59:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:59:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:59:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:59:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:59:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:59:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:59:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 16:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 16:59:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 16:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 16:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 16:59:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:00:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:00:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:00:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:00:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:00:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:00:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:00:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:00:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:00:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:01:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:01:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:01:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:01:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:01:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:01:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:01:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:01:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:01:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:01:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:01:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:01:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:02:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:02:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:02:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:02:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:02:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:02:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:02:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:02:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:02:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:03:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:03:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:03:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:03:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:03:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:03:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:03:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:03:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:04:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:04:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:04:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:04:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:04:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:04:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:04:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:04:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:05:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:05:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:05:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:05:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:05:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:05:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:05:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:05:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:05:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:05:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:05:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:05:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:06:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:06:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:06:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:06:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:06:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:06:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:06:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:07:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:07:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:07:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:07:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:07:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:07:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:07:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:07:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:08:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:08:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:08:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:08:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:08:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:08:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:08:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:08:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:08:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:08:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:08:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:08:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:09:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:09:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:09:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:09:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:09:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:09:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:09:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:10:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:10:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:10:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:10:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:10:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:10:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:10:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:10:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:10:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:10:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:10:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:10:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:11:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:11:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:11:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:11:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:11:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:11:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:11:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:11:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:11:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:11:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:11:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:11:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:12:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:12:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:12:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:12:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:12:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:12:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:12:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:12:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:12:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:12:52 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 17:12:53 rdma06 dnf[1058525]: Updating Subscription Management repositories.
Apr 15 17:12:53 rdma06 dnf[1058525]: Unable to read consumer identity
Apr 15 17:12:53 rdma06 dnf[1058525]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 17:12:54 rdma06 dnf[1058525]: Copr repo for fmf owned by psss                 2.8 kB/s | 3.3 kB     00:01
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-AppStream                                 40 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-AppStream-debuginfo                      8.6 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-BaseOS                                    62 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-BaseOS-debuginfo                          62 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-CRB-debuginfo                             58 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-HighAvailability                          59 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-HighAvailability-debuginfo                57 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-NFV                                       63 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-NFV-debuginfo                             65 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-RT                                        68 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-RT-debuginfo                              65 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-ResilientStorage                          67 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-ResilientStorage-debuginfo                66 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-SAP                                       70 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-SAP-debuginfo                             63 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-SAPHANA                                   63 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: beaker-SAPHANA-debuginfo                         61 kB/s | 168  B     00:00
Apr 15 17:12:54 rdma06 dnf[1058525]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 17:12:54 rdma06 dnf[1058525]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:54 rdma06 dnf[1058525]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 17:12:55 rdma06 dnf[1058525]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 17:12:56 rdma06 dnf[1058525]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 17:12:56 rdma06 dnf[1058525]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 17:12:56 rdma06 dnf[1058525]: Metadata cache created.
Apr 15 17:12:56 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 17:12:56 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 17:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:13:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:13:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:13:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:13:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:13:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:13:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:13:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:13:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:13:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:13:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:14:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:14:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:14:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:14:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:14:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:14:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:14:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:14:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:14:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:14:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:15:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:15:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:15:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:15:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:15:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:15:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:15:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:15:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:15:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:15:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:15:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:15:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:16:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:16:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:16:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:16:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:16:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:16:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:16:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:17:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:17:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:17:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:17:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:17:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:17:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:17:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:17:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:17:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:17:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:17:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:17:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:18:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:18:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:18:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:18:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:18:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:18:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:18:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:18:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:18:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:18:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:18:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:18:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:19:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:19:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:19:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:19:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:19:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:19:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:19:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:19:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:19:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:20:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:20:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:20:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:20:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:20:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:20:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:20:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:20:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:21:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:21:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:21:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:21:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:21:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:21:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:21:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:21:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:21:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:21:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:21:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:21:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:22:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:22:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:22:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:22:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:22:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:22:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:22:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:22:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:22:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:23:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:23:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:23:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:23:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:23:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:23:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:23:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:23:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:24:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:24:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:24:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:24:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:24:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:24:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:24:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:24:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:24:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:24:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:24:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:24:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:25:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:25:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:25:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:25:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:25:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:25:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:25:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:25:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:25:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:26:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:26:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:26:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:26:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:26:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:26:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:26:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:26:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:26:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:26:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:27:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:27:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:27:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:27:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:27:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:27:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:27:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:27:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:27:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:27:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:27:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:27:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:28:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:28:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:28:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:28:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:28:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:28:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:28:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:28:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:28:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:29:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:29:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:29:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:29:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:29:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:29:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:29:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:29:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:29:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:29:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:30:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:30:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:30:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:30:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:30:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:30:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:30:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:30:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:31:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:31:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:31:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:31:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:31:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:31:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:31:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:31:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:31:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:31:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:31:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:31:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:32:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:32:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:32:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:32:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:32:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:32:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:32:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:33:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:33:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:33:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:33:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:33:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:33:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:33:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:33:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:33:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:33:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:33:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:34:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:34:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:34:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:34:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:34:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:34:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:34:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:34:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:35:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:35:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:35:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:35:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:35:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:35:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:35:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:36:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:36:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:36:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:36:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:36:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:36:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:36:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:36:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:36:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:36:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:36:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:36:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:37:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:37:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:37:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:37:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:37:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:37:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:37:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:37:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:37:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:37:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:38:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:38:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:38:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:38:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:39:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:39:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:39:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:39:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:39:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:39:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:39:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:39:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:40:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:40:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:40:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:40:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:40:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:40:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:40:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:40:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:41:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:41:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:41:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:41:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:41:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:41:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:41:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:41:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:42:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:42:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:42:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:42:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:42:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:42:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:43:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:43:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:43:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:43:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:43:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:43:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:43:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:43:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:43:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:43:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:44:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:44:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:44:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:44:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:44:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:45:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:45:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:45:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:45:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:45:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:45:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:45:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:45:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:46:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:46:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:46:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:46:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:46:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:46:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:46:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:46:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:47:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:47:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:47:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:47:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:47:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:48:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:48:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:48:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:48:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:49:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:49:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:49:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:49:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:49:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:49:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:50:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:50:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:50:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:50:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:51:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:51:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:51:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:51:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:51:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:52:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:52:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:52:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:52:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:52:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:52:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:53:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:53:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:53:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:53:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:53:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:53:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:53:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:53:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:54:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:54:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:54:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:54:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:54:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:55:22 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:55:22 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:55:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:56:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:56:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:56:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:56:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:56:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:56:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:57:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:57:12 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:57:12 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:58:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:58:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:58:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:58:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:58:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:59:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:59:42 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 17:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 17:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 17:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 17:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 17:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:02:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:02:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:02:42 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:04:02 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:04:02 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:04:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:04:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:06:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:32 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:07:32 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:52 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:07:52 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:17:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:59:23 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 18:59:23 rdma06 dnf[1062848]: Updating Subscription Management repositories.
Apr 15 18:59:23 rdma06 dnf[1062848]: Unable to read consumer identity
Apr 15 18:59:23 rdma06 dnf[1062848]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 18:59:23 rdma06 dnf[1062848]: Metadata cache refreshed recently.
Apr 15 18:59:23 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 18:59:23 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 18:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 18:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 18:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 18:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 18:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 18:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:35:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:35:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:35:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:37:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:37:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:38:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:38:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:38:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:40:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:40:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:40:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:41:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:41:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:42:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:43:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:43:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:43:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:44:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:46:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:48:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:49:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:51:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:51:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:54:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:54:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:54:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:57:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:57:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:57:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 19:59:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 19:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 19:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 19:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 19:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:00:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:00:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:00:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:02:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:02:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:03:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:03:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:03:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:05:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:06:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:06:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:08:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:09:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:11:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:12:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:13:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:14:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:16:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:16:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:19:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:19:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:20:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:20:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:22:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:22:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:22:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:23:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:23:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:24:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:25:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:25:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:25:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:26:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:27:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:28:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:28:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:28:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:30:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:31:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:31:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:33:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:34:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:34:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:36:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:36:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:36:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:38:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:39:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:39:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:39:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:41:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:42:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:42:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:42:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:44:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:44:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:45:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:45:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:45:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:45:45 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 20:45:46 rdma06 dnf[1067149]: Updating Subscription Management repositories.
Apr 15 20:45:46 rdma06 dnf[1067149]: Unable to read consumer identity
Apr 15 20:45:46 rdma06 dnf[1067149]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 20:45:47 rdma06 dnf[1067149]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-AppStream                                 48 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-AppStream-debuginfo                       60 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-BaseOS                                    60 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-BaseOS-debuginfo                          59 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-CRB                                       58 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-HighAvailability                          61 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-HighAvailability-debuginfo                56 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-NFV                                       64 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-NFV-debuginfo                             58 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-RT                                        59 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-RT-debuginfo                              57 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-ResilientStorage                          56 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-ResilientStorage-debuginfo                59 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-SAP                                       61 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-SAPHANA                                   60 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: beaker-SAPHANA-debuginfo                         60 kB/s | 168  B     00:00
Apr 15 20:45:47 rdma06 dnf[1067149]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 20:45:47 rdma06 dnf[1067149]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:47 rdma06 dnf[1067149]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 20:45:48 rdma06 dnf[1067149]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 20:45:48 rdma06 dnf[1067149]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 20:45:49 rdma06 dnf[1067149]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 20:45:49 rdma06 dnf[1067149]: Metadata cache created.
Apr 15 20:45:49 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 20:45:49 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 20:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:47:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:47:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:47:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:48:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:48:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:49:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:50:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:50:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:50:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:51:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:53:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:55:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:56:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:58:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:58:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 20:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 20:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 20:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 20:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 20:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:01:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:01:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:01:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:04:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:04:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:04:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:06:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:07:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:07:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:07:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:09:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:09:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:10:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:10:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:10:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:12:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:13:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:13:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:15:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:16:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:18:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:19:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:20:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:21:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:23:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:23:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:26:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:26:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:27:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:27:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:29:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:29:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:29:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:30:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:30:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:31:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:32:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:32:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:32:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:33:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:34:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:35:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:35:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:35:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:37:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:38:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:38:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:40:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:41:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:41:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:43:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:43:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:43:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:45:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:46:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:46:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:46:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:48:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:49:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:49:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:49:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:51:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:51:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:52:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:52:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:52:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:54:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:54:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:54:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:55:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:55:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:56:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:57:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:57:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:57:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:58:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 21:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 21:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 21:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 21:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 21:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:00:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:02:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:03:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:05:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:05:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:08:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:08:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:08:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:11:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:11:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:11:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:13:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:14:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:14:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:14:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:16:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:16:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:17:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:17:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:17:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:19:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:20:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:20:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681611632.8383] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 15 22:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:22:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:23:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:25:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:26:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:27:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:28:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:30:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:30:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:33:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:33:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:34:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:34:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:36:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:36:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:36:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:37:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:37:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:38:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:39:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:39:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:39:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:40:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:41:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:42:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:42:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:42:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 22:44:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 15 22:44:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 15 22:44:01 rdma06 dnf[1071964]: Updating Subscription Management repositories.
Apr 15 22:44:01 rdma06 dnf[1071964]: Unable to read consumer identity
Apr 15 22:44:01 rdma06 dnf[1071964]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 22:44:01 rdma06 dnf[1071964]: Metadata cache refreshed recently.
Apr 15 22:44:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 22:44:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 22:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:44:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:45:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:45:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:45:52 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 15 22:45:52 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 15 22:45:52 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 15 22:45:52 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 15 22:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:47:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:48:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:48:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:50:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:50:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:50:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:52:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:53:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:53:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:53:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:55:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:56:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:56:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:56:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:58:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:58:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:59:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:59:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 22:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 22:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 22:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 22:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 22:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:01:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:01:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:01:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:02:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:02:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:03:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:04:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:04:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:04:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:05:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:07:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:09:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:12:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:12:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:15:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:15:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:15:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:18:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:18:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:18:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:20:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:21:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:21:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:21:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:23:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:23:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:24:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:24:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:24:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:26:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:27:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:27:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:29:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:30:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:32:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:34:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:35:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:37:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:37:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:40:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:40:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:40:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:41:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:41:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:43:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:43:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:43:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:44:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:45:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:46:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:46:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:46:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:47:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 15 23:47:01 rdma06 dnf[1074524]: Updating Subscription Management repositories.
Apr 15 23:47:01 rdma06 dnf[1074524]: Unable to read consumer identity
Apr 15 23:47:01 rdma06 dnf[1074524]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 15 23:47:02 rdma06 dnf[1074524]: Copr repo for fmf owned by psss                 2.5 kB/s | 3.3 kB     00:01
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-AppStream                                 40 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-AppStream-debuginfo                       53 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-BaseOS                                    61 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-BaseOS-debuginfo                          56 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-CRB                                       64 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-CRB-debuginfo                             63 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-HighAvailability                          66 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-HighAvailability-debuginfo                57 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-NFV                                       59 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-NFV-debuginfo                             66 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-RT                                        66 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-RT':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-RT-debuginfo                              68 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-ResilientStorage                          64 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-ResilientStorage-debuginfo                65 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-SAP                                       65 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-SAP-debuginfo                             66 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-SAPHANA                                   67 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: beaker-SAPHANA-debuginfo                         68 kB/s | 168  B     00:00
Apr 15 23:47:02 rdma06 dnf[1074524]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 15 23:47:02 rdma06 dnf[1074524]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:02 rdma06 dnf[1074524]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 15 23:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:47:03 rdma06 dnf[1074524]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 15 23:47:04 rdma06 dnf[1074524]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 15 23:47:04 rdma06 dnf[1074524]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 15 23:47:04 rdma06 dnf[1074524]: Metadata cache created.
Apr 15 23:47:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 15 23:47:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 15 23:47:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:48:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:49:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:49:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:49:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:51:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:52:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:52:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:54:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:55:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:57:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:57:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:57:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 15 23:59:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 15 23:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 15 23:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 15 23:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 15 23:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:00:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:00:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
